tjaddison.com

Half a year to complete the migration to Eleventy. What will I do with the rest of it?

Timothy Addison — Sat, 22 Apr 2023 00:00:00 +0000

After six months this blog is now running on Eleventy instead of Gatsby. In my previous post I talked about why I wanted to make the switch. Today I'll talk briefly about how I found the migration, whether the result is what I hoped for, and make a few predictions about the future of the site.

The upgrade process

Don't be fooled by the six month timing between posts. I'd suggest that with focus the migration could have been done in a long day or a weekend.

I'd like to say the delta between that long weekend and six months was purely down to limited time, but I know I spent a lot of time trying to understand how Eleventy worked and why certain decisions had been made.

Towards the end of the migration I stumbled on a few interesting blogs that helped me understand how people were using Eleventy in the wild - I never found the official docs to be helpful in this respect, and would suggest anyone looking to grok Eleventy starts by reading as broadly on the topic (through the eyes of others) as possible.

Coming from Gatsby - a very much 'batteries included' framework which is supported by a strong ecosystem of documentation - at times Eleventy felt like I wasn't smart enough to figure out what was clearly an obvious thing. The amount of time I wondered about what was the idiomatic way to put images in my posts left me convinced I'd missed a tutorial somewhere (I'm pretty sure I haven't). The learning curve felt needlessly high in many places, with the documentation gleefully presenting all the features but little guidance on when you might want to use or avoid them.

The results

The build process is faster - both locally and on Netlify, with builds on a cold cache dramatically faster. The client-side bundle has been eliminated. There is no more GraphQL involved. The number of referenced development packages (and thus npm install performance) is greatly reduced.

Navigating between pages doesn't feel as snappy as before (Gatsby pre-fetched and performed client-side navigation), which is a tradeoff I knew I'd make in exchange for eliminating the client-side bundle. I haven't noticed any other changes, and the core web vitals agree.

Happy with the current status of the project and site.

I'll caveat the above by adding that I have migrated but not authored anything yet nor had to go through any upgrades. On top of that, I've set myself a 'no tinkering until the first blog post is published' rule which means that I can't even speak to the editing experience.

Predictions for the future

Offered with probabilities I've pulled out of thin air, and something I can come back and examine in the future.

100% chance I'll publish at least one post on the Eleventy blog in the next month (as I write this draft, I feel supremely confident)
75% chance I'll complete at least two items from my 'do soon' checklist in the next three months
50% chance I'll complete at least four of them in the next three months
10% chance I'll complete all of them in the next three months
75% chance I'll publish at least two posts in the next six months
75% chance I'll do something from the 'do later' checklist in the next six months
50% chance I'll do a quarter of them in the next six months
25% chance I'll publish at least six posts in the next twelve months
5% chance I'll spend time searching for an alternative static site generator in the next twelve months, based on an observed limitation/friction with the setup

The 'do soon' and 'do later' checklists are in the project's readme file - here's the current commit on GitHub if you'd like to see what's there.

The next time you see me I will be Eleventy

Timothy Addison — Thu, 13 Oct 2022 00:00:00 +0000

After slowly completing a series of explorations into how I can migrate from Gatsby to Eleventy, I've decided to draw a line in the sand and say this will be my last post until it's done.

The rest of this post will cover why I decided to switch away from Gatsby, what I'm hoping to achieve with Eleventy, and thoughts on why I've struggled to make the switch.

History

This blog was originally built with Jekyll. At the time I knew nothing about Ruby or Liquid templates (largely unchanged today), and I made little effort to learn about the ecosystem. I was mostly happy with Hugo, though did have a pretty rough time when I tried to get my feet wet with Ruby - I like to think it was because running Ruby on Windows was a pain at the time, but more honestly I wasn't invested in learning that language.

A few years later I was writing React recreationally and came across Gatsby. Their zero-touch image optimisation spoke to a problem I'd started noticing with my site (where I did nothing to optimise images). That, plus the ability to work with a Static Site Generator (SSG) in a language I was comfortable with (JavaScript) convinced me to trial a migration, and I remember it being relatively painless to see demonstrable performance gains for my site. I migrated.

I wrote about my motivations to move in the post migrating from Jekyll to Gatsby, which includes criteria I applied when selecting Gatsby.

I was happy with this setup for a while, though did find the hit & miss plugin quality and GraphQL everywhere (even places it had no business being) were detractors from the experience.

Build Timeouts and Upgrades

At some point in late 2020 my Netlify builds started to time out (getting up to twenty minutes on the free tier before being terminated). Image processing accounted for ninety percent of this, which was surprising given the site had around a hundred images. Luckily Netlify was emitting warnings that there was a Gatsby cache plugin available that allowed assets to be shared between builds.

This experience planted the seed to start looking elsewhere, a twenty minute build time for a site as small as mine was unreasonable.

I'm not without blame - I've done nothing to ensure the source images are well-sized and compressed, which I'm certain would cut down on build times regardless of the image processing pipeline I use.

The experience that committed me to move was attempting the Gatsby v3 to v4 major upgrade. A modest collection of Gatsby plugins caused me to hit a number of issues with breaking changes on the plugins, and one edge case that caused builds to never complete (which was later fixed, but this was an awful experience for a core plugin that handled images).

I never completed the major upgrade, and my site is currently stuck at Gatsby 3.9.1. I'm sure some issues I experienced are fixed by now, but I'm convinced the only future this ecosystem has to offer me is more pain with dependency upgrades.

Lessons Learned and Looking Ahead

Reviewing the criteria that I used when deciding on the last migration, the only thing Gatsby failed to deliver on was low/zero ongoing cost. A new requirement came from a more nuanced view of what Fast means to me, encouraging me to eliminate the giant bundles I was shipping with the site (a truly static site).

The other requirement should be a simple set of dependencies that are easy to extend. JavaScript is an incredibly capable language, and so many plugins and their deep tangle of dependencies are unnecessary.

Dependencies and the software supply chain are something I've been spending a lot of time wondering about with a view to wondering how durable a given software or tool is going to be. I covered it briefly in a pocket app with no dependencies.

After initially considering and rejecting Eleventy due to the requirement of learning a new template language (Liquid or Nunjucks), I finally came back to it after accepting a learning curve was reasonable, and in the case of Nunjucks the templating language was popular.

The other candidate I spent a long time considering was NextJS. After a few prototypes it was rejected as needlessly complicated for my needs, and categorically failed on the dependencies test.

Not Migrating

As of early 2022 I'd decided there was no future in Gatsby, and that Eleventy was the right answer. And it is now October 2022 and I'm writing a post to assist in motivating me to get it done. What's caused the delay?

A big part of the delay is that life has been happening, as it is wont to do.

The other part was working on the blog became an energy drain - it used to be something I enjoyed extending and tweaking, and now every interaction with the setup makes me wish I was using literally anything else.

To combat this I tried to make forward progress with proof of concepts - things like ensuring I could get images working. These explorations produced the post processing images linked from frontmatter with Eleventy but didn't get the blog any closer to migrated. I'm convinced this was mostly a procrastination tactic, and I could have learned everything I needed experimenting in a branch or fork of the site.

And so that's what I'm going to do now.

This post is the swan song of the Gatsby v3 blog. The next post will be coming from the Eleventy edition.

Processing images linked from frontmatter with eleventy-img to use in meta tags

Timothy Addison — Wed, 31 Aug 2022 00:00:00 +0000

Given a markdown file (/blog/hello-world/index.md) with frontmatter that looks like this:

---
shareimage: "small.png"
---

How can we process that image with eleventy-img plugin and render a meta tag:

<!-- note the generated filename is a content hash-width combination -->
<meta property="og:image" content="/blog/hello-world/xfO_genLg4-600.png" />

This post will cover the steps needed, and assumes a basic familiarity with Eleventy.

Despite tonnes of great documentation of the various features of Eleventy and its plugins, the lack of an example that showed processing images via frontmatter left me struggling with this as a newcomer to the ecosystem.

Setup

In addition to @11ty/eleventy you'll need @11ty/eleventy-img. The code below assumes you're using Nunjucks as the template language. Finally, we've got the following file structure (note the images are in the same folder as the markdown file):

/blog
  /hello-world
    index.md
    giant.png
    small.png

Processing the image

To copy the image to our output folder (along with resizing and any other processing we want to do) we'll be creating a shortcode:

// .eleventy.js
const Image = require("@11ty/eleventy-img");

async function shareImageShortcode(src) {
  // src might be small.png - taken from frontmatter
  const { url } = this.page;
  // url might be /blog/hello-world/
  const imageSrc = "." + url + src;
  let metadata = await Image(imageSrc, {
    widths: [600],
    formats: ["jpeg"],
    urlPath: url,
    outputDir: `./_site/${url}`,
  });

  const data = metadata.jpeg[0];
  // data.url might be /blog/hello-world/xfO_genLg4-600.jpeg
  // note the filename is a content hash-width combination
  return data.url;
}

module.exports = function (eleventyConfig) {
  eleventyConfig.addNunjucksAsyncShortcode(
    "shareImageUri",
    shareImageShortcode
  );

  return {
    markdownTemplateEngine: "njk",
  };
};

The code above is making a few assumptions:

We are creating a jpeg image (broadest support - we can't use multiple sources)
We're making that image 600 pixels wide (we can't use a srcset)
The output directory is _site

The contents of this.page are well documented and are available in all shortcodes

The shortcode we've created is shareImageUri which we can now use in a template.

Using the shortcode in a template

We can use the shortcode in a template (which will need to be one that can render into the final head tag) like this:

<!-- part of default.njk -->
<head>
  <meta>
  {% if shareimage %}
    <meta property="og:image" content="{% shareImageUri shareimage %}" />
  {% endif %}
  </meta>
</head>

Because of the data cascade the frontmatter value shareimage will be available. If you want a fallback image you can use an {% else %} block to render a hardcoded alternative. You'll probably want to put that somewhere in the root of your site (e.g. /img) rather than generating the same placeholder in every post's folder (which using the shortcode we created would do).

And that's all there is to it!

What did I miss

As a total newbie to Eleventy I'm sure I've missed some obvious stuff. If you can point me in the right direction on any of the below please mail me eleventy-feedback at this blog's domain.

Rather than hardcoding _site is it accessible somewhere? I can see that the outputPath starts with that but I'd need to assume the output path was a single folder only.
This all seemed fairly simple in hindsight, and the documentation has all the bits needed to get it working. Is there an image tutorial I missed that explains this, or I am just spoiled coming from a world where Gatsby did a pretty good job at processing everything as an image?
Working with local images using relative file paths was a little tricky both here and with rendering images in markdown posts (another post!) - is there a go-to tutorial/document/starter that covers this? Some of my path manipulation code feels like it could have been easier?

Migrating my digital garden from WikiLens to Foam, and taking it private

Timothy Addison — Sat, 09 Jul 2022 00:00:00 +0000

I've previously talked about how much I loved the workflow that GistPad (and more recently WikiLens) enables for curating a digital garden. With the addition of github.dev browsing and editing are now possible from a browser - something I called out as missing two years ago. Given the slickness of the setup, what's prompted a change? In short: data gravity.

Too many gardens

Having multiple digital gardens has long been a pain point - specifically as the digital garden I curate with my family grows, there are more instances when daily notes straddle the public and the private. In some cases the hee-hawing over whether to write one note in each, a split note, or come up with some interesting cross-reference system has been distraction enough to not write anything down at all. Working with the garage door up comes with some hidden costs it seems.

I'll still have blog posts as a very public outlet, and in the future consider exploring ways to mark notes in a private garden as public. More on which in a moment.

Not enough... gardening tools?

The metaphor has got away from me a little, I'll admit.

When collapsing the two digital gardens together I decided to take another look at how Foam had come along (after dismissing it as a little too early two years ago). I was pleased to see it delivers on its vision of providing 'just-enough' tooling, but no more than that. A couple of items I was extremely happy to see were templates, nested tags, and frontmatter support. Coupled with a vibrant community and a low switching cost, I decided to make the move.

Migrating

I migrated from a wiki that worked with WikiLens. These steps may work for other markdown-based wikis, but I've not tried them.

Start by modifying the .vscode/extensions.json file to recommend Foam rather than the extension you were using:

{
  "recommendations": [
-    "lostintangent.wikilens"
+    "foam.foam-vscode"
  ]
}

Create an empty .vscode/foam.json file to tell the Foam extension that this is a Foam workspace:

{}

To get the daily page working I updated the .vscode/settings.json to include the following:

{
+  "foam.openDailyNote.directory": "Daily/",
+  "foam.openDailyNote.titleFormat": "dddd, mmmm d, yyyy"
}

This uses the fulldate format and defaults to creating the page in the Daily folder. The default filename is still yyyy-MM-dd.md, though that can also be customised by a setting.

I strongly encourage you to read the Foam documentation, and clone the sample to see all the features in action. The migration steps here only cover a subset of the capabilities of Foam.

Differences

One of the only differences I've noticed is that WikiLens uses the page title for links (as defined by the H1 in the document), whereas Foam uses the page name (based on the filename). As an example, links to daily pages required updates for their links:

- Last time I worked on this (see [[August 18, 2020]])...
+ Last time I worked on this (see [[2020-08-18]])...

Early thoughts

Beyond the fact I never have to think about which digital garden to put a daily note in (hooray!), not much has really changed. This is a great result, and further confirms my suspicion that markdown based note-taking is likely to be durable all the way through to the long now.

One thing I've found surprising is how little content I've been able to organically discover that discusses Foam (how people are using it, extending it, migrating to it). Part of this could be the name (Foam and foambubble are challenging search terms to get lost amongst), though it could also be the community is not publishing this content and it's all in Discord/GitHub issues/Twitter.

What next?

For now the focus will be on ensuring any friction introduced by collapsing the two digital gardens into one will be minimal. So far this is mostly a question of disambiguation. On my public (only for me) wiki I had a Personal Blog page which clearly meant tjaddison.com. In the private (for our family) wiki that could mean any one of a number of personal blogs - thankfully renaming files automatically updates all references.

Introducing Foam has also allowed us to revisit some of our tagging strategies - we used to use hyphens to create parent/child structures, but with Foam we get native support for parent/child tags.

Once we've worked through any teething issues I'm very interested to explore what can be done with templates and frontmatter to create a much richer experience for some of the things we log in our digital garden (think books, restaurants, trips).

What are the costs of an engineering career progression framework?

Timothy Addison — Sat, 30 Apr 2022 00:00:00 +0000

I've rarely seen the costs / return on investment (ROI) mentioned when discussing career progression frameworks. It turns out there are lots of interesting costs to be aware of, and some of them can be significant. If you're planning to roll out a framework, being aware of the costs as well as the benefits will increase the chance of your framework coming down on the right side of the ROI calculation.

I was inspired to dig into this by Yvonne Lam's thread what needs do levels serve, which contains the insightful question "what problems are orgs choosing to have when they introduce levels". The related question of "how else can could those needs be served" is also fascinating, but we'll stick to costs for now.

Visible Costs

Starting with the costs that are easiest to see, these are costs that will be incurred and are directly linked to building and leveraging a framework.

The first cost is building the framework. Even with a wealth of inspiration to draw from the first draft of the framework will take several hours. Iterating towards a version you're ready to roll out will take dozens of hours from multiple people, with the cost increasing proportional to the ambition and complexity of your framework (how many roles? how many levels?).

Just like any other product you build, the initial cost pales next to the cost to maintain the framework. Technology definitely won't stand still, and hopefully neither will your company. There will be bugs and inconsistencies in the framework that will only be uncovered as you start to use it. A rule of thumb is probably whatever it took to build you'll be spending in annual maintenance costs if you want to keep it relevant.

A well built and maintained framework will need to be used to actually deliver any value - which means you're going to have to spend time training people to use and apply the framework. If you're going to use your framework for career growth as well as recruiting, this is potentially going to mean you'll also need to align your recruiting evaluation with your framework. This is going to be both a large upfront investment as well as ongoing work for every employee and interview loop.

I don't see any of these costs as avoidable if you want to build an effective framework. The only suggestion I'd offer is ensure you focus on 'good enough' at each step and assemble the smallest possible group of people to own the framework. The heuristic I've most often used is if adding another person doesn't meaningfully increase the diversity of the group, don't make the group any larger.

Now we're done with the visible and fairly innocuous costs, let's dig into the tricky ones.

Invisible Costs

The largest cost of a framework is mostly invisible, and is related to incentives. If the framework does not perfectly align with delivering customer value, employees will be incentivised to make decisions based on personal progression over customer value. This ranges from individuals making decisions about the work they'll pursue, to the managers who will need to make decisions about which projects to give to certain individuals to help them secure a promotion. Incentives are tremendously powerful, and so this is a cost to ignore at your peril.

Pursuing promotion is a perfectly rational motivation. Attempts to paint it as an employee failing - when hundreds of thousands of dollars in a raise can be on the line - typically comes from someone in a position of overwhelming privilege.

Frameworks will also create many opportunities for people to be dissatisfied with their position in the framework. The costs here can range from temporary demotivation, to the loss of an employee, to the inability to hire an employee whose only issue is a perceived (or real) mis-titling. In an extreme situation you may be faced with a candidate who would be a stellar performer in the role, but you cannot hire them as they don't fit into your framework.

These two costs are significant enough that for me they eclipse the visible costs, and should be front and center when building and rolling out a progression framework.

The final cost I'll cover is the illusion of objectivity. Being more objective is a common reason cited to build a framework, but most (all?) frameworks have just moved the unconscious bias around. Almost every promotion will be tied to the opportunities afforded to a given individual, which will come back to bias and inclusion. Do you think a phenomenal framework would have changed Susan's very strange year at Uber? Anywhere you catch yourself discussing something that is more meritocratic remember the paradox of meritocracy.

So, no framework?

I don't see an alternative to not having a framework, though I'd like to see us as an industry propose and test alternatives. Although Powerful makes a very compelling case for no framework/levels, it glosses over the significant diversity & inclusion challenges a framework can assist with. That alone is reason enough for me to continue to advocate for a framework, even with the caveat around the illusion of objectivity.

Writing a simple Pocket app in Node.js with no dependencies

Timothy Addison — Thu, 31 Mar 2022 00:00:00 +0000

As I've started using Pocket to track articles I've read/would like to read, I wanted to build a workflow that I could use when writing my Link Roundups. At the same time, I've been wondering how productive you can be with no dependencies. A simple Pocket app to download my saved articles and write them to a file felt like a good test.

The source code is available on GitHub as PocketDumper - the rest of this article focuses on the thought process rather than how it works.

This is my personal experience for a set of tools that has limited requirements. On larger projects, or in your tools - you may find the dependencies, frameworks, etc. to be invaluable. They have a place and the no dependencies approach is definitely not appropriate everywhere!

Why no dependencies?

Building tools to make my life easier is something I've been doing for a long time. As those tools have aged they occasionally require moderate to heroic efforts to get them working again.

A few months ago I moved from Windows to macOS and I've had a lot of tools that were built against platform-specific toolchains or dependencies suddenly demand a lot of effort to get them working. If switching OS is rare, I'd rank encountering issue with dependency updates on almost anything built against a modern web toolchain/stack to be common. The blog post I wrote about updating babel-eslint to @babel/eslint-parser remains the most popular post on my site by a large margin - I don't think I'm alone in suffering from upgrade hell.

The tools that have required the least TLC are shell scripts (luckily for me recent versions of PowerShell run cross-platform), though the developer experience is not ideal - especially when they start to grow into miniature applications rather than tools.

Containers seem to be the logical answer here, but I've experienced both compatibility issues with the move to Apple Silicon and the - in my mind - larger issue that the first-run time can be atrocious (time and energy spent to run what amounts to a trivial script).

If I was a stronger C/C++ programmer I think I'd be writing all of my tools in that language. Although I'm not building my tools for the long now, I'd wager that if they were in C++ they'd probably work just fine in a hundred years. Instead, I've settled on JavaScript and Node.js.

How can you say no dependencies and then pick Node? Haven't you heard about node_modules?

It has been a very long time since I've started a node project and not immediately installed multiple packages (or started from a template/framework such as create-react-app or Next.js). Even with Next's efforts to reduce dependencies there's still 150+ dependencies installed along for a brand new project. On top of that, I can't think of the last project I created where I didn't start off with one of either TypeScript or Babel plus a bundler.

But what if we gave all that up, what are we left with?

Well, as of Node 17.5+ - it turns out we can get quite a lot done.

My Setup

I've cheated a little on the no dependencies front by leveraging Volta to provide me with access to nodemon in my toolchain. I'm making the assumption that JavaScript and Node are going to be around for a long time, and so global tools like nodemon will be accessible. Even if Volta disappears, I'm confident npm install -g will still be an option.

To get type checking without TypeScript I'm using VS Code with a jsconfig file. This doesn't give me anywhere near the same experience as I get with a full TypeScript project, but it definitely gives me enough to miss common errors - type annotations are still possible with JSDoc comments.

{
  "compilerOptions": {
    "target": "es2021",
    "module": "es2022",
    "checkJs": true
  }
}

I'm also taking advantage of the fact that fetch is included with Node from 17.5+ (behind a flag) to remove the need for node-fetch or similar packages. It's due to be included without the flag in Node 18.

node --experimental-fetch index.js

// pocketApi.js
const getRequestToken = async function (consumerKey) {
  const response = await fetch(endpoints.GetRequestToken, {
    method: "POST",
    headers,
    body: JSON.stringify({
      consumer_key: consumerKey,
      redirect_uri,
    }),
  });
  /// elided...
};

My index.js file also takes advantage of top-level await which has been available since Node 14.8 as long as the file or project is set to ESM rather than CJS:

// index.js
import { getUserData } from "./util.js";

let { AccessToken, ConsumerKey, Since = 0 } = await getUserData();

Along with this there are now a lot of promise-based APIs available that mean callback's are no longer needed for things like readline:

import * as readline from "node:readline/promises";
import { stdin as input, stdout as output } from "node:process";

const rl = readline.createInterface({ input, output });
await rl.question("Press enter once you have authorized the application\r\n");

These are just a small set of the enhancements that have landed in Node (and the broader JavaScript ecosystem) in the last few years. It's a significant amount of effort to get up to speed but looking back at tools I've authored 2, 3, or even 5 years ago - they could all be much smaller and take on less dependencies.

I'm not sure I'd go back and rewrite them (excepting some dependency upgrade torture), but for future tools I'll certainly check what's available in native node/modern version of JavaScript, before reaching for a dependency.

What's next?

The only thing I've really felt that was missing was a good test runner, although I'm fairly confident that for the kind of tools I'm building I'll get pretty close with a little bit of assert. I've also not had a look at the ecosystem to see if using a test runner via the Volta toolchain might also work for me.

Update: A little research shows that Node 18 is going to get a built-in test-runner. What wonderful timing!

I was initially sceptical I'd get anything useful done with no dependencies. After completing this exercise I reviewed the tools I've found enduringly useful and they all involve a fairly small set of operations - filesystem interactions, http calls, and manipulation of in-memory data structures. With ES2022 and Node 18 (particularly if Temporal lands) the surface area of 'native Node' has never looked so compelling.

Benchmarking .NET’s Dictionary.TryGetValue by key length and runtime

Timothy Addison — Mon, 28 Feb 2022 00:00:00 +0000

How much faster is it to use a shorter key for a string-keyed dictionary? And how does that vary based on .NET 6.0 vs. .NET Framework 4.8? The avalanche of performance improvements in the runtime had me convinced the newer runtime would be faster, but by how much, and what order of magnitude was the impact of key length on performance (nano, micro, or milliseconds)?

We'll start with the results:

Benchmark Setup

The source code is on GitHub. The method I've benchmarked is:

public int TryGetValue(string key) => _dictionary.TryGetValue(key, out var value) ? value : default;

The benchmark supports running that for keys of various length (I did 1 to 201 in increments of 25) over 5 runtimes, leveraging BenchmarkDotNet's arguments source.

The csproj file to support multiple runtimes looks like this:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFrameworks>net6.0;net5.0;netcoreapp3.1;net48;netcoreapp2.1</TargetFrameworks>
    <LangVersion>7.3</LangVersion>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.13.1" />
  </ItemGroup>

</Project>

Benchmark Results - All Runtimes

The runtimes cluster into a slow group (.NET Framework 4.8, .NET Core 2.1) and a fast group (.NET Core 3.1, .NET 5.0, .NET 6.0). On the machine I used (and given the benchmark run durations) I'm attributing the deltas between the fast runtimes to noise, as repeated runs would sometimes change the ordering. This isn't surprising as we're talking about a handful of nanoseconds between them in many cases, which is a couple of processor cycles at best.

On the machine I ran the benchmarks on the results show a cost of around 10-15ns at a key length of 1 character, increasing by 1ns/character for the slow group, and 0.5ns/char in the fast group. These are incredibly small numbers (2-4 processor cycles on the CPU I benchmarked on), and they're worth looking at alongside more common latency numbers. Using the .NET 6.0 runtime event at a key length of around 200 characters the cost of that lookup is about the same as accessing RAM.

Takeaways

Outside of code on a very hot path this probably doesn't matter (though if in doubt, benchmark and trace to be sure!). Being thoughtful with key length will improve performance (I've seen examples where namespaces were used to prefix keys for uniqueness - and some namespaces can be long). If getting every last drop of performance is essential, storing data in a field/property is going to be faster (a shared dictionary is often something that is convenient to use).

When it comes to field vs. property there really isn't any difference (I'd be curious to see if anything differs in .NET 6, but not curious enough to run that benchmark).

If you'd like to read more about low-level details on memory performance (while we're in the realm of nanoseconds it's helpful to understand what various activities 'cost'), this Stack Overflow post has some great answers/comments/links.

Early thoughts on switching to Apple silicon and macOS from Windows

Timothy Addison — Mon, 31 Jan 2022 00:00:00 +0000

I've been a Windows power user (personally and professionally) for over 20 years. In December 2020 I switched my primary machine to an M1 MacBook Pro and macOS Monterey, from a Surface Book 3 and Windows 10. After two months with the setup I'm sharing my early thoughts on the transition.

My workload consists primarily of using the Microsoft Office suite (someone once jokingly told me moving deeper into management meant switching an IDE out for Word/Outlook - it wasn't a joke, and they forgot to mention Excel!), with some recreational coding (Node/React) as well as professional SQL Server development. I spend a lot of time consuming written content on the web, and even more time in video calls (using Teams) on workdays.

The specific devices I'm comparing are a 13.5" Surface Book 3 i7/32GB/512GB SSD and a 14" M1 MacBook Pro 10-core/16GB/512GB SSD. At the time of writing the Mac is about $400 cheaper than the Surface Book.

OS

After a relatively painful (yet short) learning curve on shortcuts, I'm mostly past the 'who moved the cheese' phase of learning the OS. I didn't find any particularly useful cheat-sheets or articles, mostly reading documentation (checking carefully for the version of the OS it applies to) or looking up shortcuts in the menu. Command+Shift+? is extremely useful for this (search within menus).

One item I've found myself missing is accessing menu items via the alt key - the solution to which appears to be either learn the shortcut (if the menu item has one), or use the menu search. Browsing the menu with the keyboard feels clunky and I find myself reaching for the trackpad a lot.

Native window management (especially with multiple monitors) is poor, and after trying to live with the native options (which ended up with a lot of trackpad usage) I installed Rectangle (brew install --cask rectangle) and have never looked back.

Speaking of Homebrew - it's how I wish I'd installed everything. I used Chocolatey off and on for Windows (and think winget may have a bright future), but the breadth of packages and support for homebrew blows them away.

Overall I'm still far more productive at OS tasks in Windows.

Terminal and Shell

My setup was [Windows Terminal] and [PowerShell 7] (aka pwsh or PowerShell Core). iTerm2 (brew install --cask iterm2) is a fantastic terminal, and the installation instructions for PowerShell 7 on macOS were easy to follow. Once installed, you need to set pwsh as the default shell in iTerm2 (which pwsh will tell you the path to the binary, and in iTerm preferences you want Profiles > Default > Command > Custom Shell).

I had some issues getting homebrew working in pwsh, which I fixed by adding the following line to my PowerShell profile (edit via code $PROFILE):

$Env:PATH += ":/opt/homebrew/bin"

The documentation in general for Apple Silicon is harder to find, and PowerShell on Apple Silicon even more so.

With homebrew working in PowerShell I got oh-my-posh installed easily, on top of which I added Z. I've not yet really finished my exploration of chezmoi, and I'd like to see what a repeat experience looks like with that (or even sharing configuration between Windows and macOS).

Office Suite

My prior exposure to Excel on a Mac was a miserable one a few years ago, assisting a friend with what should have been a quick pivot and some formulae and it turned out...there were a lot of missing features. Fast forward to today and I can only count one missing feature that I wanted to use (The Lambda function).

The other tools (Word, Outlook) I can't say I've noticed any feature gaps.

Re-learning keyboard shortcuts has been the only pain-point, though not enough to have a serious impact on my productivity. My Excel usage is nowhere near that of someone who participates in the Financial Modelling World Cup - I only know enough to be mildly dangerous.

I'm about as productive on the Mac as I was on Windows.

Development

For the bulk of my development work (which is mostly recreational - Node/React), this has been nothing but an improvement. Both in perceived and measured performance (I ran a lot of Measure-Command to check install/build times) everything is snappier. Despite the Mac being cheaper than the Surface Book, for almost everything I could measure the Mac was anywhere from 2x to 10x faster.

As most of my day is spent in some way interacting with JavaScript (VS Code, reading anything on the web) the performance difference is palpable.

The Mac is hands-down the better experience, completing similar activities on my Windows machine feels sluggish.

SQL Server Development

No good news here. If you want to do any meaningful SQL Server development you're probably going to need a Windows device. While you can install SQL Edge on the Mac for trivial development, that doesn't have much overlap with the kind of work I do (the list of unsupported features is long).

Even Azure Data Studio (brew install --cask azure-data-studio) is a poor replacement for SQL Server Management Studio (SSMS).

I'm lucky that we leverage Azure Virtual Desktop (AVD) at work which allows me to run an instance of SSMS on my Mac via application virtualization. This does require a lot of infrastructure to support, but is almost indistinguishable from running SSMS on a Windows machine.

If I didn't have access to AVD I'm not sure what I'd do here - probably develop against an Azure SQL Database with Azure Data Studio, and feel bad about it.

The Surface Book is far superior here.

Overall Impressions

I can't imagine switching back. Although the build quality of the Surface Book was good, the Mac is excellent. Any concerns I had about the keyboard and typing speed were quickly dispelled (I tested myself about the same on both devices).

Battery life on the Mac is exceptional - I can get through a full working day of 8-10 hours use, whereas I'd need the power adapter on the Surface Book. I've never heard the fans on my Mac, unlike the Surface Book where even video calls would cause them to spin up.

Even though I can't imagine switching back, if Windows performance could approach M1 performance - I'd give that a try.

Don't mistake their kindness for empathy

Timothy Addison — Fri, 31 Dec 2021 00:00:00 +0000

When considering someone for a management role I've seen people mistake kindness for empathy. Having a kind manager is definitely better than having an unkind manager, but to effectively manage people you should be looking for an empathetic manager.

I've seen this happen on multiple occasions - in some cases an individual confuses their own kind behaviour for empathy! The rather low baseline for engineering managers probably plays a part here, where a table-stakes trait like kindness sets you apart as a potential manager. In addition to that, we continue to put people into management positions who are incapable and/or unqualified for the role because we have no idea what to look for.

This doesn't mean that kindness isn't something I look for - it's actually one of the first things I want to see when evaluating people management capabilities:

Are they kind?
Are they empathetic?
Are they curious about their team?

But before we get too far, we need to look a little closer at that 'kind' word.

You can't always be kind

Although advice like "be unfailingly kind" and "keep your feedback actionable, specific, and kind" sounds great, Chelsea Troy points out the problem with that statement, namely:

...it assumes that the feedback giver has control over whether their feedback is perceived as kind. And they don’t. from Transcript from lecture: Giving and receiving feedback

This goes beyond feedback, and extends to all interactions with multiple parties - kindness is really in the eye of the beholder, and so the best you can do is act with kind intent.

Anyone who can share examples of where they've experienced a gap between their intent (be kind) and the impact (not perceived as kind) does well here. The best candidates will be able to share examples of where they've changed the way they interact to minimise that gap (a great example Chelsea shares is: not giving someone feedback if they're actually looking for validation).

Being able to act with kind intent requires empathy, which we'll come onto next.

Empathy isn't only for kind people

The reason I'm looking for kindness before empathy is because being empathetic isn't locked away down the nice-person skill tree. I'd argue that some of the most hurtful people are incredibly empathetic and leverage that to cause significant harm. Testing for empathy without controlling for kindness can let brilliant jerks through, so I'd never start with empathy.

Empathy is often presented alongside sympathy and compassion:

Sympathy - I am sorry you're going through that
Empathy - I can imagine what it must feel like to be going through that
Compassion - What can I do to help?

This framing sticks with people, such that being empathetic benefits from the positive associations of sympathy and compassion, despite empathy being somewhat orthogonal to those traits.

Assuming you've verified kind intent, checking for empathy typically starts with the question "what were they thinking?". This can either be used in made-up scenarios (you request a meeting to discuss progress on a project which isn't going well), but are best when you're discussing real scenarios (delivering a performance review, promotion, termination, mediating a heated debate).

The best candidates will be able to answer a wide variety of follow ups:

What were they trying to achieve?
What were they thinking but didn't feel they could say, and why?
What was their biggest concern?

But empathy alone isn't why they'll excel here, because empathy works best when paired with a deep knowledge of the individual.

Curious about their team

Every new manager starts out with almost zero knowledge of their new team. No amount of empathy or kind intent can make up for this, and so curiosity and a commitment to learn more about their team is crucial.

I've found this is best evaluated alongside empathy with the follow-up question "and why did you think that?". I'm not sure if there is a clear distinction on good vs. great as having any answer is a good sign (most candidates stumble at empathy).

Interestingly, I've seen a mix of both talent and skill at work here. Some people are more naturally talented (curious) and will always look to learn more. Other people aren't, but are able to make up for that deficit with a system/approach they apply. In either case, one common factor is consistency. If you don't show up and spend time with your team every week, it doesn't matter how good your system is/how talented you are - you'll never learn anything.

Aside: Projection and Bias

Be prepared for this line of questioning to be the first time someone has really introspected about why they thought something - I'd consider this a fairly advanced skill. As a consequence, you may end up in a coaching session on projection and/or bias.

As everyone is unique we're bound to attempt to pattern match behaviours - and the set of behaviours we're most familiar with is our own. When you ask someone why an engineer really wanted to learn a new technology, a common answer (when you really dig) is "because that's what I was interested in doing at that stage in my career". Projecting your own motivations onto someone else isn't necessarily wrong (and with experience you gain a larger catalogue of patterns to match against), but awareness you're doing it is important.

Projection is less likely when you're dealing with someone who is different, and so here is where you need to watch out for bias. As an example, consider a discussion about an engineer who always volunteers to take minutes, and when you ask "why do you think she does that?" the candidate replies "because she is naturally talented at organizing things". Would your answer have changed if the engineer was male? Did you ever probe to validate your assumption? Could this have been a case of glue work automatically getting assigned to the only female engineer on the team?

These kind of conversations (which are really great coaching opportunities) happen more often in regular 1:1s, but I have ended up here in a few interviews. The first interview where the conversation turned into a discussion on unconscious bias I wasn't particularly ready for (and neither me or the candidate enjoyed that segment). If you're going to pursue this line of questioning, be ready to have the full conversation.

If you'd like to read more on bias I'd strongly encourage you to work through all the links at the start of Why your efforts to make your company inclusive aren't working.

In summary

To be a great people manager you want to look for someone who:

Acts with kind intent
Is empathetic
Is curious about their team/colleagues

More generally, these are useful skills to work on in whatever role you have, irrespective of whether you are or ever want to be a manager. They're also skills that can be improved with practice - so even if you don't consider yourself empathetic, try asking yourself "what were they thinking" after the next interesting interaction you have.

Interesting Links - November 2021

Timothy Addison — Tue, 30 Nov 2021 00:00:00 +0000

Links are broadly categorised into Development, Management and leadership, and the catch-all Miscellaneous.

Development

Capturing my frustrations on what an npm-outdated looks like for my Gatsby-powered blog, Ru Singh's waving goodbye to static websites and more does make the 'DIY' option look like a poor choice. Every time I think about switching to Wordpress (or something else) I run through the framework I used last time I switched my blog (back in 2019) and always land on static sites. For now.

One important feature for me is cost - ideally free, and stack on a budget is a good place to look for inspiration. Another feature is performance, and Speedlify can let you track performance over time (though Lighthouse scores are not without their challenges).

Another month goes by and the GitHub issues for optimize images during next build...maybe inches towards being implemented? I'll probably have to ignore this and rely on the default next/image component, though I'm not a fan of being locked into Vercel (as a free host). When the time comes to migrate, the next-mdx-remote library is probably how I'll get the MD/MDX used (there's even a starter template).

Finally, if you find yourself wanting a reminder of what the relentless pursuit of pragmatism looks like, these quotes from Rasmus Lerdorf (creator of PHP) are worth reviewing. As a taster - "I actually hate programming, but I love solving problems."

Management and Leadership

Scaling security across an organization is hard - Clint Gibler's presentation How to 10X your security is an excellent starting point (and probably covers stuff that most orgs won't even get to in their end-state). Clint also maintains the excellent infosec newsletter tldrsec.com, which reduces the firehose of infosec news to a ...slightly smaller firehose?

A common criticism of most business books I read is 'that could have been a long-form article'. Imagine my delight when I discovered The Kool Aid Factory, a site full of zines (in this instance: long-form articles) that cover various aspects of organizational design. In addition to having a tonne of content that will get you thinking, the site's filters allow you to focus on what matters to you (though I ended up devouring everything - no filters for me).

From articles that resisted becoming books, to a presentation that resisted becoming a book - Coordination Headwind - How organizations are like slime molds covers the challenge of keeping an organization nimble as it grows.

I've always felt a little uneasy with the advice that states you should 'never be the smartest person in the room', as it always felt like a crutch to lean on when you don't want to have a conversation about the values of a diverse team (smart sometimes being a codeword for cishet-white-male). A much better rebuttal can be found in don't surround yourself with smarter people, which is a long but rewarding read that switches out smart for differently free.

I don't think the dust has settled on the future of work (at least I hope it hasn't, and that we're still in a phase of the pandemic that doesn't fully represent the new normal), but I do think there is a lot we can learn about the good & bad of remote from the natural experiment COVID-19 provided. The negative impact on bridging/weak/cross-team ties is what I was most interested in exploring in The effects of remote work on collaboration among information workers. I definitely felt that being fully remote 'impacted serendipity', but that is much more rigorously formulated and explored in the paper. No solutions on offer here though - I wonder if more radical organizational changes are needed to succeed with a remote org (rather than taking the existing organizational structure and simply removing everyone from the office)?

Regardless of remote or in-person, articles on how to have an honest one on one, fantastic threads on soliciting feedback as a manager or the kitchen-sink of resources (How to Lead People and Be a Manager) are well worth revisiting in what is turning out to be a very challenging few months/years/new normal.

I've also ordered the book Become an effective software engineering manager, though I do wonder when I'll get round to reading it. Meanwhile, James has released a new book on effective remote work, and I've still not finished going through his back-catalogue of his excellent posts on engineering management. An ever-expanding anti library is a great problem to have.

Miscellaneous

People

In addition to recently following the below, I also highly recommend their archives.

I first encountered Derek Sivers thanks to his Now page (mine is still a TODO), but I stuck around for the thoughtful and clear writing on a wide range of topics.

I can't remember where I first encountered Rachel's writing, but I suspect it was a link to one of the (many, excellent) war stories she has shared. I found myself nodding along an awful lot, and appreciate someone sharing the unvarnished truth on how dysfunctional work can be.

For longer, thought-provoking reads (also on a wide range of topics), the writing of Scott Alexander never disappoints. Seeing Scott highlight and respond to some of the better comments in follow-up posts is a unique touch, and it helps to see the material discussed from different viewpoints.

Finally, something that has so far defied my attempts to track via Feedly - Gwern Branwen's website. I'm not sure what to call it - digital garden, long content, evergreen blog - whatever it is, the thought process behind the site really resonated with me. In addition to deep and engaging material, the UX of the site is truly excellent (and quite necessary given all the fascinating links and references).

Tools

For editing tables in markdown (or CSV for that matter), the grid table editor supports operations like adding/removing/moving columns, which is not much fun to do manually.

Another great tool I've started using recently to enhance a text (e.g. markdown) documentation is monodraw.

Only after hacking together a script to scaffold new blog posts I discovered plop, an incredibly versatile 'micro-generator framework' for node - the combination of inquirer and handlebars is pretty versatile.

I've recently started using a Mac in addition to my Windows machine, and one of the first things I installed was PowerShell core. While tweaking my shell (starting with Oh My Posh) I gave the z PowerShell command a try, and I can't believe I didn't try it sooner, the amount of time I spend navigating directories has dropped dramatically.

Well-being

A notable gap in the data I collect (see quantified self) is mental health - will I remain 'young minded' as I age, and will I be free from any mental illness? For the former question I've looked at Ryff Scales and some tests on Your Morals, and for the latter it seems positive mental health correlates with a decline in the incidence of mental illnesses (see the paper Change in Level of Positive Mental Health as a Predictor of Future Risk of Mental Illness). A few open questions I have are what can be done to improve mental health, and how reliable is self-evaluations.

Typing

Right now I'm only considering it, but moving to a split keyboard and learning a new keyboard layout might help maximise my ability to type effectively over time (and maybe boost my keys left). In anticipation of switching, I've started to capture some basic typing speed results with monkeytype.

I'd like to consume more research on the topic, so far the only paper I've completed was to confirm something I was already fairly certain of - tactile feedback matters for typing speed and accuracy (A Study of Touch Typing Performance with Keyclick Feedback).

Deploy an Azure AD protected App Service Website with Pulumi

Timothy Addison — Sat, 23 Oct 2021 00:00:00 +0000

This post will walk through how to use Pulumi to deploy an Azure App Service application secured with Easy Auth. Under the default configuration only authenticated users will be able to access the application, without any custom code (easy auth places an authentication/authorization middleware in front of your app). If you'd like to jump straight to the code you can see a full example project on GitHub.

I'll be using Azure Active Directory in this example, though easy auth also supports Microsoft (personal account), Google, Facebook, Twitter, and OpenID Connect.

I'm assuming you've already completed the Pulumi Azure pre-requisites (or similar), and have the appropriate permissions in your tenant and Azure subscription. When we're done we'll have deployed a Pulumi stack with the following resources:

Creating the project

Start by creating the application, and adding the AzureAD package we'll need to create the Azure AD application registration.

pulumi new azure-csharp `
  --name easyauth-webapp `
  --description "azure ad secured app" `
  --stack dev `
  --config azure-native:location=eastus

dotnet add package Pulumi.AzureAD

Note the example defaults to eastus - you can set this to whatever region you want, I've included it here so the script will complete without any prompts.

We next need to update the contents of the pulumi.dev.yaml file to contain a few additional config items. Paste the following into the file:

config:
  azure-native:location: eastus
  azure-native:subscriptionId: UPDATE_ME
  azure-native:tenantId: UPDATE_ME
  easyauth-webapp:tenantId: UPDATE_ME
  easyauth-webapp:ownerId: UPDATE_ME
  easyauth-webapp:siteName: UPDATE_ME
  easyauth-webapp:appRegistrationName: UPDATE_ME

You can set siteName and appRegistrationName to whatever you want, though for simplicity's sake I'd suggest using the same item. As an example I might pick easy-auth-azure-ad.

The subscriptionId and tenantId should be set to the appropriate target's for your Azure app service and Azure AD application registration, respectively. I recommend setting the ownerId to your user's id, otherwise you may find you don't have the permission to modify or delete the application registration after it has been deployed.

The following commands may be helpful in retrieving these values (requires the Azure CLI):

# Get your user's id
az ad signed-in-user show --query objectId

# List all subscriptions (and their tenant) that you have access to
az account list

Note tenantId is set twice as I couldn't figure out how to access the azure-native:tenantId via configuration, and it is needed both to set the default tenant for the application registration deployment, and to construct the token issuer URI.

Deploy the website (no security)

We'll next create the website we want to deploy. We're going to use the run from ZIP package functionality to deploy the contents of the wwwroot folder. Create that folder and add some content to the index.htm file:

<!-- wwwroot/index.htm -->
<html>
  <head>
    <title>A very secure app</title>
  </head>
  <body>
    Hello EasyAuth with Pulumi!
  </body>
</html>

Now we can deploy this file to Azure with Pulumi. Modify the MyStack.cs file to contain the below code, which has been adapted from the Pulumi Function Stack example:

// MyStack.cs
using System;
using Pulumi;
using Pulumi.AzureAD;
using Pulumi.AzureAD.Inputs;
using Pulumi.AzureNative.Resources;
using Pulumi.AzureNative.Storage;
using Pulumi.AzureNative.Storage.Inputs;
using Pulumi.AzureNative.Web;
using Pulumi.AzureNative.Web.Inputs;

class MyStack : Stack
{
  public MyStack()
  {
    var config = new Pulumi.Config();
    var tenantId = config.Require("tenantId");
    var ownerId = config.Require("ownerId");
    var siteName = config.Require("siteName");
    var appRegistrationName = config.Require("appRegistrationName");

    var rg = new ResourceGroup($"RG-{siteName}");

    var storageAccount = new StorageAccount("storageaccount", new StorageAccountArgs
    {
      ResourceGroupName = rg.Name,
      Kind = "StorageV2",
      Sku = new SkuArgs
      {
        Name = SkuName.Standard_LRS,
      },
    });

    var appServicePlan = new AppServicePlan("appserviceplan", new AppServicePlanArgs
    {
      ResourceGroupName = rg.Name,
      Kind = "App",
      Sku = new SkuDescriptionArgs
      {
        Tier = "Basic",
        Name = "B1",
      },
    });

    var container = new BlobContainer("zips", new BlobContainerArgs
    {
      AccountName = storageAccount.Name,
      PublicAccess = PublicAccess.None,
      ResourceGroupName = rg.Name,
    });

    var blob = new Blob("appservice-blob", new BlobArgs
    {
      ResourceGroupName = rg.Name,
      AccountName = storageAccount.Name,
      ContainerName = container.Name,
      Type = BlobType.Block,
      Source = new FileArchive("wwwroot"),
    });

    var codeBlobUrl = SignedBlobReadUrl(blob, container, storageAccount, rg);

    var app = new WebApp("app", new WebAppArgs
    {
      Name = siteName,
      ResourceGroupName = rg.Name,
      ServerFarmId = appServicePlan.Id,
      SiteConfig = new SiteConfigArgs
      {
        AppSettings = {
          new NameValuePairArgs{
              Name = "WEBSITE_RUN_FROM_PACKAGE",
              Value = codeBlobUrl,
          }
        },
      }
    });

    this.Endpoint = app.DefaultHostName;
  }

  // From https://github.com/pulumi/examples/blob/master/azure-cs-functions/FunctionsStack.cs
  private static Output<string> SignedBlobReadUrl(Blob blob, BlobContainer container, StorageAccount account, ResourceGroup resourceGroup)
  {
    return Output.Tuple<string, string, string, string>(
        blob.Name, container.Name, account.Name, resourceGroup.Name).Apply(t =>
    {
      (string blobName, string containerName, string accountName, string resourceGroupName) = t;

      var blobSAS = ListStorageAccountServiceSAS.InvokeAsync(new ListStorageAccountServiceSASArgs
      {
        AccountName = accountName,
        Protocols = HttpProtocol.Https,
        SharedAccessStartTime = "2021-01-01",
        SharedAccessExpiryTime = "2030-01-01",
        Resource = SignedResource.C,
        ResourceGroupName = resourceGroupName,
        Permissions = Permissions.R,
        CanonicalizedResource = "/blob/" + accountName + "/" + containerName,
        ContentType = "application/json",
        CacheControl = "max-age=5",
        ContentDisposition = "inline",
        ContentEncoding = "deflate",
      });
      return Output.Format($"https://{accountName}.blob.core.windows.net/{containerName}/{blobName}?{blobSAS.Result.ServiceSasToken}");
    });
  }

  [Output] public Output<string> Endpoint { get; set; }
}

We can now deploy the site and verify it has worked as intended:

pulumi up --stack dev

curl (pulumi stack --stack dev output Endpoint)

Securing the site

To configure Easy Auth we first create an Azure AD application registration. In this example I'm specifying AzureADMyOrg which restricts access to the tenant the application registration is deployed in. I'm also adding a RedirectUri that points at the Easy Auth middleware of the deployed site. A password is needed to use as a client secret (the web application being the client in this case).

Once the application registration is created we can add WebAppAuthSettings to our site. The example specifies no anonymous access (using RedirectToLoginPage), and connects the site to the application registration using the ClientId and ClientSecret (password).

Paste the below code just after the this.Endpoint... code in MyStack.cs:

// MyStack.cs
// After this.Endpoint = app.DefaultHostName;

var adApp = new Application("ADAppRegistration", new ApplicationArgs
{
  DisplayName = appRegistrationName,
  SignInAudience = "AzureADMyOrg",
  Owners = new[] { ownerId },
  Web = new ApplicationWebArgs
  {
    ImplicitGrant = new ApplicationWebImplicitGrantArgs
    {
      IdTokenIssuanceEnabled = true
    },
    RedirectUris = new System.Collections.Generic.List<string> { $"https://{siteName}.azurewebsites.net/.auth/login/aad/callback" }
  }
}
);

var applicationPassword = new ApplicationPassword("appPassword", new ApplicationPasswordArgs
{
  ApplicationObjectId = adApp.Id,
  DisplayName = "Client secret for web app"
});

var allowedAudience = adApp.ApplicationId.Apply(id => $"api://{id}");

var authSettings = new WebAppAuthSettings("authSettings", new WebAppAuthSettingsArgs
{
  ResourceGroupName = rg.Name,
  Name = app.Name,
  Enabled = true,
  UnauthenticatedClientAction = UnauthenticatedClientAction.RedirectToLoginPage,
  DefaultProvider = BuiltInAuthenticationProvider.AzureActiveDirectory,
  ClientId = adApp.ApplicationId,
  ClientSecret = applicationPassword.Value,
  Issuer = $"https://sts.windows.net/{tenantId}/v2.0",
  AllowedAudiences = new[] { allowedAudience },
});

We can now update the site, and if we try to access the endpoint we'll notice it is no longer available over http. From the command line we can't get much further than this, but in a browser we'll get redirected to complete the login flow and access the site.

pulumi up --stack dev

# Redirect to HTTPS
curl (pulumi stack --stack dev output Endpoint)

# Access denied
curl "https://$(pulumi stack --stack dev output Endpoint)"

Cleaning up

You can remove all the resources with the following command:

pulumi destroy --stack dev

And then remove the stack from the Pulumi console with:

pulumi stack rm dev

Notes

This example is using WebAppAuthSettings rather than WebAppAuthSettingsV2 due to a known bug that prevents it from working. Once this bug is fixed I recommend updating to v2, as the classic experience is due to be deprecated from the Azure portal.

While building and debugging this I found the Microsoft Graph Explorer to be helpful. I managed to waste a good half hour until I realized the important distinction between the application registration's object id, and the application registration's application id. Both were needed in this example!

Finally, an observation that it's fairly common to have the permission to create application registrations, but if you fail to specify yourself as an owner you won't be able to edit/delete it. If you're not developing in a test tenant you might need to speak to IT/Security to clean up some failed attempts (speaking from experience...).

Run Pester 5 tests in Azure Pipelines and publish test results

Timothy Addison — Thu, 30 Sep 2021 00:00:00 +0000

The brevity of the Migrating from Pester v4 to v5 documentation belies just how much has changed - something that took a fair bit of trial and error for me was getting Pester 5 test results published in an Azure Pipelines run.

Assuming you have your tests located in a tests folder, the pipeline definition needed to run the tests and publish the results is below.

I could have used Invoke-Pester -Path tests, but I frequently find tests which use relative paths, and so invoking them from the correct folder is easier.

# /pipelines/azure-pipelines.yml
pool:
  vmImage: windows-2019

steps:
  - task: PowerShell@2
    displayName: "Run Pester tests"
    inputs:
      targetType: "inline"
      script: |
        Set-Location ./tests
        Invoke-Pester -CI
      ignoreLASTEXITCODE: true

  - task: PublishTestResults@2
    inputs:
      testResultsFormat: "NUnit"
      testResultsFiles: "**/Test*.xml"
      failTaskOnFailedTests: true
      testRunTitle: "Validate Task Files"

The -CI argument to Invoke-Pester will save test results in an xml file (NUnitXML compatible) in the same folder. It will also set the exit code of the process to the number of failed tests, which is problematic as any non-zero exit code will abort the pipeline by default. To ensure our pipeline continues when tests fail, we set ignoreLASTEXITCODE on the PowerShell task. To stop the pipeline in the presence of failed tasks we use the failTaskOnFailedTests property of the Publish Test Results task.

In the example below, I'm running the Test-FolderTask function of tSqlScheduler, and as you can see when I break the build on purpose, the tests fail 😊.

Interesting Links - August 2021

Timothy Addison — Tue, 31 Aug 2021 00:00:00 +0000

With the advent of tab groups I can now claim to have 'only a few tabs' open, yet have dozens (maybe more) links pile up for later review. I'm not sure that's an improvement 🤔.

Links are broadly categorised into Frontend, Development, Management and leadership, and the catch-all Miscellaneous.

Frontend

Webmention Analytics

Add visibility to your site's webmention implementation with Webmention Analytics. In addition to being something you can fork and deploy pretty quickly, it also demonstrates how you can build on top of your webmention data. Assumes you're using webmention.io.

TypeScript Deep Dive

If you prefer learning via reading (and the official Typescript Handbook wasn't your style), I found the TypeScript Deep Dive to be an accessible and much more practical introduction (includes topics like project setup, getting up and running with a React app). In addition to the basics, it includes real-world usage tips, a style guide, and a deep dive into the TypeScript compiler internals.

DOM Events

DOM Events is a fantastic visual exposition of the browser's DOM event system that needs to be seen to be appreciated (click the Dispatch button!). I can't think of a better way to teach/understand the model - a great UI that is both simple, informative, and yet feature-packed.

React from scratch

This in-depth and enlightening three-part series (starting with Climbing Mount Effect) lays out some of the problems React's architecture solves, and then extends that architecture with some very different solutions.

If you've never read the details on why some of React's limitations exist (one-way data flow, rules regarding hooks, effects) - this piece is well worth the effort to work through. And if you do understand those limitations, this piece shows another way of thinking about them - really helped consolidate my mental model of React.

If you're curious about building React from scratch, I'd recommend the fantastic Build your own React. Not only is the content great, but the presentation is fantastic.

Javascript

Looking forward to hopefully seeing the proposal for JSON modules make it into the standard. I've got a whole bunch of JSON being imported in various utility tools, and if it could fail-fast if it's no longer JSON (for any reason), so much the better.

import json from "./foo.json" assert { type: "json" };

And now looking back, after reading authentication in React apps it is hard to unsee the many times I have made the same mistakes over and over. At least next time I need to write an app I've got a solid pattern to follow, and maybe one day I'll refactor some existing code into the pattern (hahaha...).

Looking forward again, I'm already confident that Next.js + Tailwindcss are my go-to choices for building. Hosting is a little trickier. While I've been very happy with Netlify for static sites, because Next.js integrates so tightly with Vercel that's probably an upcoming change I'll make. Nothing has yet dislodged firebase/firestore as the go-to web accessible data store to use...until I started reading about Supabase. And given it's actually SQL (hooray!) I'm even more interested than I was before. Reading through an end to end tutorial has convinced me this is worth exploring more.

Another missing piece of my puzzle is how to make calls to that backend - in most cases I'm using a straightforward useFirestore hook that has some home-rolled caching, but after reading through practical react-query I'm wondering if that might not be a better fit. I started the series with react-query as a state manager and was hooked (no pun intended!), and promptly went back to the beginning to complete the series.

Development

Local-first Software

Data longevity is something I've been thinking about a lot recently, especially when applied at a personal level (can I leverage this data when I am 20, 30 years older?). The Local-first software article covers this as well as 6 other concerns that point towards a very different kind of software.

In addition to the principles, the Automerge library demonstrates that collaborative applications don't have to be incredibly challenging to implement (leave that to the library authors).

I also noted Replicache that provides a hosted solution in the same space.

Data Longevity - Datasets

While thinking about data longevity I realized that I care far more about programmatic consumption of data, more than merely reading it - and so my focus is on datasets. The US Library of Congress recommendation for datasets suggests:

Formats using well known schemas with public validation tool available
Line-oriented, e.g. TSV, CSV, fixed-width
Platform-independent open formats, e.g. .db, .db3

I've been focusing on JSON Lines as my personal choice recently - the fact it's human readable is a huge plus, and it's also very close to the interchange format for most sites I'm building (JSON round-trips well over the web!).

However, after reading (and frankly having my mind blown) hosting SQLite databases on GitHub pages I might reconsider using it instead of 'JSON databases'.

Environment setup

One rough edge I've found with codespaces so far is environment setup - and the docs for customizing your codespace sent me down the rabbit hole of environment configuration.

René-Marc's dotfiles were my jumping-off point, and in addition to discovering scoop I also found the rather formidable chezmoi. A lot to digest here (even the how-to is huge!), but it does look like the comprehensive solution that will let me work between Windows and macOS and linux.

But there's a lot of work to do to get there...

Security beyond on-premise

While the tier model worked well for on-premise networks, it wasn't a good fit for a hybrid or cloud-native org (or basically any environment where you need to have Zero Trust). If you're familiar with the tier model then the Privileged Access Model provides a mapping from old to new.

Azure

The discussion of how the Urlist app is built covers a lot of ground, and along the way hits Front Door, Cosmos DB, CNAME flattening, Azure Functions and more. It's eminently practical and also covers cost (which is never present on architecture diagrams), and left me with a better understanding of how some of the different tools fit together.

When deploying all of those apps to Azure, perhaps rather than inflicting ARM templates on yourself you could instead use the new Bicep language. It's an improvement, but I don't know if it's enough of an improvement over something like Pulumi to be worth learning.

I've still not really leveraged durable functions, but found the new backend announcement to be particularly interesting. In particular for the SQL backend I was curious how they implemented the workflow schema, as I've had some painful experiences with workflow/orchestration inside of SQL Server before; one of the notes at the bottom confirmed what I had seen before:

In many cases, the database will be the primary performance bottleneck.

But with the option to use the Netherite provider there is a faster, cheaper (albeit more setup required) option. The FASTER technology that is used by Netherite is...fast!

And finally - tying together Security and durable functions is Cloud Katana. Love the work Microsoft security are doing to let people improve and iterate on security safely (knowing where to start is hard, this is a fantastic onramp).

Access control, explained

RBAC like it was meant to be is the best introduction to access control I've ever read. Tailscale is an extremely impressive product, and the documentation/blog posts are just as impressive.

Management and Leadership

Growing Inclusive Behaviours

One way to start is defining those behaviours, and Chelsea Troy's Rubric for Evaluating Team Member's Contributions to an Inclusive Culture is still my go-to post/resource in this space. Highly recommended for everyone (not just managers).

Up Next for organisations?

While everybody is thinking about more immediate changes related to remote work and the 'new-normal', what longer-term (think 2031) changes are already happening? How organisations are changing covers a lot of ground (and remote vs. in-office is fairly pivotal question that is covered).

Engineering career growth

Looking at something like Dropbox's career framework you could be fooled into thinking that if you could just define the ladder (Dropbox carefully don't call it a ladder, but it is) then you're half-way towards solving 'growth'. The ladder framing (level 1...N) is common, and I'm now wondering if it may be harmful when it comes to what I call terminal levels (the combination of individual/company/team means growth is not possible or required).

The specific problem I have with the ladder metaphor is that nobody climbs halfway up a ladder and stops - everyone wants to get to the top/get off the ladder. This can lead to conversations with a senior engineer (who will never make staff) and the discussion can end up fixating on 'why not staff'. How can we instead focus on framing 'continued excellence as a senior engineer'?

Acknowledging the changing requirements of a role is important (if you level as senior now, invest nothing in personal development, would you still level as a senior in 5 years? 10 years?), and perhaps that is a way to frame a conversation about sustaining excellence. This isn't purely technical either - reading through career development for engineering managers, how many of those points were you considering 5/10+ years ago? Gergely's tweet hits a similar note - and I'd say what makes those EM's unique is the ability to stay executing in the top N% for that role.

One thing I've found valuable in defining the senior+ roles is it allows you to identify projects/behaviours you would like to see other roles (either adjacent like EMs, or more junior for the same role) execute. The list of those is long, here are a few I've enjoyed:

Even if you don't want to make these activities your focus (EM/staff engineer), making participation in them part of your role/though process may be part of how you sustain excellence.

Miscellaneous

Orbit

Spaced repetition (think Anki) taken to the next level. What if we could get much, much more from reading, by changing the format in which we author and then consume the text? See Tools for Transformative Thought for the background, Orbit for an introduction, or jump right into the Orbit documentation for implementation details. This is no less than an attempt to reimagine the humble written word as a tool for learning, and I think it's worth keeping an eye on.

My debugging hero

I didn't think I had a debugging hero, but then I read another of Bruce Dawson's posts (this time on Arranging Invisible Icons in Quadractic time) and then realised that actually, I do!

Agency

As a (relatively) freshly minted parent I'm now starting to think about education, and reflecting back on my experiences I can say that I'd like to do better. Reading The most precious resource is agency struck a chord, mainly in that outside of the social value of school, I'm not sure how valuable our current system is compared to what it could be.

Benchmarking different versions of NuGet packages

Timothy Addison — Fri, 16 Jul 2021 00:00:00 +0000

Last month I looked at benchmarking different runtimes to see what impact updating to the latest version of .NET might have. But what if you're curious about package updates too? BenchmarkDotNet has us covered there too - it allows you to configure your benchmarks to run against multiple versions of the same package. We can also leverage that functionality to benchmark packages that provide different implementations of the same abstract class or interface (e.g. DbConnection as implemented in Microsoft.Data.SqlClient and System.Data.SqlClient).

If you'd like to jump right into an example project with all the bells and whistles check the SqlClientUpdate benchmark on GitHub.

The rest of this post will walk through:

Benchmarking two different versions of a package
Benchmarking different sets of package
Benchmarking different packages that implement a common abstract class
Benchmarking all of the above with different runtimes

Benchmarking two different versions of a package

We'll start by quickly scaffolding an app and adding a benchmark for connecting to SQL Server with System.Data.SqlClient and Dapper. First of all create the app and install the packages - we're going to be looking at changes in data access, so we'll install versions of those packages that correspond to March 2019:

dotnet new console
dotnet add package BenchmarkDotNet
dotnet add package Dapper -v 1.60.6
dotnet add package System.Data.SqlClient -v 4.6.0

It's important to install the lowest version of the package you want to benchmark, as if you install the latest version any attempt to restore a lower version will fail.

Next we'll modify our Program.cs to use BenchmarkSwitcher to launch our benchmarks:

// Program.cs
using BenchmarkDotNet.Running;

public class Program
{
  public static void Main(string[] args) =>
    BenchmarkSwitcher.FromAssemblies(new[] { typeof(Program).Assembly }).Run(args);
}

And add a very minimal benchmark implementation to SqlClientBenchmark.cs:

// SqlClientBenchmark.cs
using BenchmarkDotNet.Attributes;
using Dapper;

public class SqlClientBenchmark
{
  private static string CONNECTION_STRING = "server=localhost;integrated security=sspi";

  private System.Data.Common.DbConnection _connection;

  [GlobalSetup]
  public void Setup() => _connection = new System.Data.SqlClient.SqlConnection(CONNECTION_STRING);

  [GlobalCleanup]
  public void Cleanup() => _connection.Dispose();

  [Benchmark]
  public void Execute() => _connection.Execute("SELECT 1");
}

If we wanted to see what the impact of upgrading System.Data.SqlClient to the latest version would be, we need to use a Config:

// SqlClientBenchmark.cs
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using Dapper;

[Config(typeof(Config))]
public class SqlClientBenchmark
{
  private class Config : ManualConfig
  {
      public Config()
      {
        var baseJob = Job.Default;

        AddJob(baseJob.WithNuGet("System.Data.SqlClient", "4.6.0"));
        AddJob(baseJob.WithNuGet("System.Data.SqlClient", "4.8.2"));
      }
  }
// ...

If we run the benchmark we'll now get one run for each version of System.Data.SqlClient specified.

dotnet run -c Release --filter * --join

Benchmarking two different sets of package versions

We can also pass a list of packages, so our code in Config becomes:

AddJob(baseJob.WithNuGet(new NuGetReferenceList() {
      new NuGetReference("System.Data.SqlClient", "4.6.0"),
      new NuGetReference("Dapper", "1.60.6"),
  }));

AddJob(baseJob.WithNuGet(new NuGetReferenceList() {
      new NuGetReference("System.Data.SqlClient", "4.8.2"),
      new NuGetReference("Dapper", "2.0.90"),
  }));

The benchmark will now compare the March 2019 versions against the July 2021 versions of both Dapper and System.Data.SqlClient.

Benchmarking different packages that implement a common abstract class

We'll now consider what would happen if rather than upgrading to the latest version of System.Data.SqlClient, we instead upgraded to the latest version of Microsoft.Data.SqlClient. We'll create a single abstract class BaseBenchmark which will hold all the benchmark definitions, and then two classes which implement BaseBenchmark, one for each of the *.SqlClient packages we want to test.

You don't have to use abstract classes and multiple implementations, and if you're only benchmarking a single method it is probably overkill (you might want to consider clipboard inheritance instead 😊).

Our base class contains all common code:

// BaseBenchmark.cs
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Jobs;
using Dapper;

public abstract class BaseBenchmark {
  protected static string CONNECTION_STRING = "server=localhost;integrated security=sspi";
  protected static Job BaseJob = Job.Default;
  protected System.Data.Common.DbConnection _connection;

  [GlobalCleanup]
  public void Cleanup() => _connection.Dispose();

  [Benchmark]
  public void Execute() => _connection.Execute("SELECT 1");
}

And each implementation contains a different Config, and connection initialisation:

// SystemDataBenchmark.cs
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;

[Config(typeof(Config))]
public class SystemDataBenchmark : BaseBenchmark
{
  private class Config : ManualConfig
  {
    public Config()
    {
      AddJob(BaseJob.WithNuGet(new NuGetReferenceList() {
            new NuGetReference("System.Data.SqlClient", "4.6.0"),
            new NuGetReference("Dapper", "1.60.6"),
        }));

      AddJob(BaseJob.WithNuGet(new NuGetReferenceList() {
            new NuGetReference("System.Data.SqlClient", "4.8.2"),
            new NuGetReference("Dapper", "2.0.90"),
        }));
    }
  }

  [GlobalSetup]
  public void Setup() => _connection = new System.Data.SqlClient.SqlConnection(CONNECTION_STRING);
}

// MicrosoftDataBenchmark.cs
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;

[Config(typeof(Config))]
public class MicrosoftDataBenchmark : BaseBenchmark
{
  private class Config : ManualConfig
  {
    public Config()
    {
      AddJob(BaseJob.WithNuGet(new NuGetReferenceList() {
            new NuGetReference("Microsoft.Data.SqlClient", "3.0.0"),
            new NuGetReference("Dapper", "2.0.90"),
        }));
    }
  }

  [GlobalSetup]
  public void Setup() => _connection = new Microsoft.Data.SqlClient.SqlConnection(CONNECTION_STRING);
}

We'll also need to install the new NuGet package:

dotnet add package Microsoft.Data.SqlClient

Running the benchmark now compares all three options (baseline, System.Data upgrade, Microsoft.Data upgrade):

Benchmarking all of the above with different runtimes

So what if we wanted to also see what impact updating from .NET Framework 4.8.2 to .NET 5.0 would have? We've been running our previous benchmarks with .NET 5.0, but in early 2019 we'd have been using .NET Framework 4.8.2. To see what difference that makes we'll update our System.Data benchmarks to look like this:

// SystemDataBenchmarks.cs
using BenchmarkDotNet.Environments;
// ...
    public Config()
    {
      var oldPackages = BaseJob.WithNuGet(new NuGetReferenceList() {
            new NuGetReference("System.Data.SqlClient", "4.6.0"),
            new NuGetReference("Dapper", "1.60.6"),
        });

      var newPackages = BaseJob.WithNuGet(new NuGetReferenceList() {
            new NuGetReference("System.Data.SqlClient", "4.8.2"),
            new NuGetReference("Dapper", "2.0.90"),
        });

      AddJob(oldPackages.WithRuntime(ClrRuntime.Net48));
      AddJob(oldPackages.WithRuntime(CoreRuntime.Core50));
      AddJob(newPackages.WithRuntime(ClrRuntime.Net48));
      AddJob(newPackages.WithRuntime(CoreRuntime.Core50));
    }
// ...

This will compare the old packages on both runtimes, as well as the new packages on both runtimes. We'll also try both runtimes for the Microsoft.Data package update:

// MicrosoftDataBenchmarks.cs
// ...
    public Config()
    {
      var packages = BaseJob.WithNuGet(new NuGetReferenceList() {
            new NuGetReference("Microsoft.Data.SqlClient", "3.0.0"),
            new NuGetReference("Dapper", "2.0.90"),
        });

      AddJob(packages.WithRuntime(ClrRuntime.Net48));
      AddJob(packages.WithRuntime(CoreRuntime.Core50));
    }
// ...

We'll also need to update the csproj file to allow us to target multiple frameworks:

- <TargetFramework>net50</TargetFramework>
+ <TargetFrameworks>net48;net5.0</TargetFrameworks>

And invoking the benchmark now requires us to specify which framework we want to execute the host process with:

dotnet run -c Release -f net48 --filter * --join

The result of which is six different benchmarks (and so if we had a lot of methods to benchmark we might be here for quite a while):

If you'd like to see an example that contains multiple benchmark definitions (still focusing on SqlClient) I'd encourage you to check out the SqlClientUpdate benchmark on GitHub.

All the benchmark results here are to demonstrate the techniques only - don't take away anything from the results. If you are interested in benchmarking data access from SQL Server I'd encourage you to use a longer duration (Job.VeryLongRun is what I use), use a dedicated SQL machine rather than localhost, and monitor the SQL instance to ensure any slowdowns aren't on the SQL side. This is easier said than done!

Run benchmarks using multiple runtimes with BenchmarkDotNet

Timothy Addison — Tue, 08 Jun 2021 00:00:00 +0000

It's never been easier to run benchmarks against multiple runtimes with BenchmarkDotNet. Maybe you're checking your performance improvements benefit all runtimes targeted by your library code (that may support multiple target frameworks). Maybe you're curious to see if the relentless march of runtime improvements will benefit some of your existing code? Whatever the reason, with a couple of lines of code you can generate benchmarks like this:

I first saw this technique explained in the performance improvements in .NET 5 post which made it easier to understand than the toolchains documentation. The rest of this post doesn't introduce anything not already covered in the post/docs, but might be easier to digest.

Setup

We're going to compare the performance of String.StartsWith to Span.StartsWith under:

.NET Framework 4.8 (download developer pack)
.NET Core 3.1 (download SDK)
.NET 5 (download SDK)

We'll then create a new console app for our benchmarks:

dotnet new console --name spanBench

This bootstraps a new console application in the spanBench folder.

Project File

We're going to make three changes to our spanBench.csproj file:

Add the additional target frameworks (using the appropriate Target Framework Moniker for each, see target frameworks)
Add the BenchmarkDotNet dependency
Conditionally add the System.Memory package when targeting .NET Framework 4.8 (the System.Span type is included in the runtime from .NET Core 2.1)

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFrameworks>net5.0;netcoreapp3.1;net48</TargetFrameworks>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.13.0" />
  </ItemGroup>

  <ItemGroup Condition=" '$(TargetFramework)' == 'net48' ">
    <PackageReference Include="System.Memory" Version="4.5.4" />
  </ItemGroup>
</Project>

Benchmark Code

The code is configured to use BenchmarkSwitcher, which allows you to specify which benchmarks to run via the command line, or interactively (if no filter argument is passed). We've also added the MemoryDiagnoser attribute to collect information about allocations. The benchmarks are fairly simple and will each be executed with two different arguments.

using System;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using BenchmarkDotNet.Diagnosers;

namespace SpanPerf
{
    [MemoryDiagnoser]
    public class Program
    {
        static void Main(string[] args) => BenchmarkSwitcher.FromAssemblies(new[] { typeof(Program).Assembly }).Run(args);

        [Arguments("http://www.google.com")]
        [Arguments("https://www.google.com")]
        [Benchmark]
        public bool String_StartsWith(string str)
        {
            return str.StartsWith("https://");
        }

        [Arguments("http://www.google.com")]
        [Arguments("https://www.google.com")]
        [Benchmark]
        public BotDetails Span_StartsWith(string str)
        {
            return str.AsSpan().StartsWith("https://".AsSpan());
        }
    }
}

Running the benchmark

With everything in place the below command will run all 12 benchmarks (2 methods x 2 arguments x 3 runtimes):

dotnet run --configuration Release --framework net48 --runtimes net48 netcoreapp31 netcoreapp50 --filter * --join

The first two arguments are for the dotnet run command:

Use the release configuration (never benchmark with debug!)
Build using the .NET Framework 4.8 surface area

The final three arguments are forwarded to BenchmarkDotNet's Run method:

runtimes specifies the runtimes we want to benchmark with
filter in this case specifies every benchmark in the assembly
join outputs all results at the end, rather than after each individual benchmark

Publishing a self-contained single-file .NET 5 executable

Timothy Addison — Sat, 29 May 2021 00:00:00 +0000

Assuming you want to target Windows:

dotnet publish -c Release -r win-x64 --self-contained true -p:PublishSingleFile=true -p:IncludeNativeLibrariesForSelfExtract=true

And if you want to reduce the size (taking advantage of trimming):

dotnet publish -c Release -r win-x64 --self-contained true -p:PublishSingleFile=true -p:IncludeNativeLibrariesForSelfExtract=true -p:PublishTrimmed=True -p:TrimMode=Link

If you'd like some background and an explanation of those options then read on.

Background and Motivation

I build a small benchmarking tool (SQLDriver), and after updating to .NET 5 recently I realized that anyone using it may need to install an updated runtime. I'd previously made an effort to keep the executable as easy to download and use as possible (using ILMerge to bundle dependent assemblies into the executable), so wondered how easy things were with .NET 5?

Self-Contained

A self-contained .NET application is one that doesn't rely on there being any shared components on the target machine (such as the .NET runtime). This feature has actually been around for several years, and the only downside is that it can end up publishing a lot of additional files along with the app (which also means a simple console app can be 60MB+).

Getting this to work requires a single flag:

dotnet publish -c Release -r win-x64 --self-contained true

Read more on the Microsoft docs site: Self-contained publishing.

Single-File

Rather than dozens (or hundreds) of files, this option publishes only a handful of files, with the rest being unzipped into memory when the app is launched:

dotnet publish -c Release -r win-x64 --self-contained true -p:PublishSingleFile=true

The reason there isn't a single file (which is what you might expect!) is that only managed DLLs are bundled into the executable, and the native binaries (part of the .NET runtime) are left as separate files. You can have the native binaries bundled by specifying the following:

dotnet publish -c Release -r win-x64 --self-contained true -p:PublishSingleFile=true -p:IncludeNativeLibrariesForSelfExtract=true

In .NET Core 3.x the PublishSingleFile option also bundled native binaries. The IncludeNativeLibrariesForSelfExtract option is new wih .NET 5, where the default is to not bundle the native binaries.

Read more on the Microsoft docs site: Single-file deployment and executable.

Trimming

The final step for me was to enable trimming. Available only for self-contained apps, this feature allows bundling of only the assemblies that are used (rather than the entire .NET runtime):

dotnet publish -c Release -r win-x64 --self-contained true -p:PublishSingleFile=true -p:IncludeNativeLibrariesForSelfExtract=true -p:PublishTrimmed=True

We can go one step further and remove unused code from assemblies (rather than whole assemblies only):

dotnet publish -c Release -r win-x64 --self-contained true -p:PublishSingleFile=true -p:IncludeNativeLibrariesForSelfExtract=true -p:PublishTrimmed=True -p:TrimMode=link

The downside to trimming (including the more aggressive link option) is that there are certain scenarios where the build-time analysis may incorrectly identify code as unused, which will result in a runtime error. The docs call out components that can cause trimming problems - and in my case the application was very simple and also very easy to test for correctness.

Read more on the Microsoft docs site: Trim self-contained deployments and executables.

Conclusion

As a result of the self-contained, single-file, and trimming options I was able to get SQLDriver down to a single ~18MB executable that doesn't require anything to be installed on the target machine.

Getting a summary of cspell errors with PowerShell

Timothy Addison — Fri, 30 Apr 2021 00:00:00 +0000

After configuring cspell for my blog, I was confronted with over 600 errors to work through. Looking through the list it was clear that in many cases I only needed to add some words to the cspell.json workspace dictionary (though in some cases I just couldn't spell). Rather than going through each file and playing whack-a-mole I decided to aggregate all the errors and rank them:

This told me how many times the 'error' showed up, and in how many files. My thinking here was that if the word was ubiquitous then I should probably add it to the cspell.json file, and if it only showed up in one file then that file might be a better place.

This is personal preference, and mainly driven by the fact that technology can be a bit of an alphabet soup of acronyms, and as I write fairly broadly a huge custom dictionary may let typos slip through. In cases where it was an English word I'd always add it to the workspace dictionary, and in some cases where the word appeared in multiple files I'd still opt for per-file overrides (if the word was a niche technical term that could look like a typo).

So, how does the above command work?

PowerShell to the rescue

We start by capturing the output of the cspell command:

$out = npx cspell content/**/*.md*

# $out Contains one row per error:
# C:\src\blog\content\blog\2020\12\analyze-your-create-react-app-bundle-size-without-ejecting\index.md:82:70 - Unknown word (gatsbyjs)
# C:\src\blog\content\blog\2017\04\checking-sql-agent-job-ownership-with-pester\index.md:17:40 - Unknown word (msdb)
# ...snip...

We can leverage the excellent ConvertFrom-String command to pull the various items out by constructing a template:

$template = @'
{FileName*:C:\src\blog\content\blog\2020\12\analyze-your-create-react-app-bundle-size-without-ejecting\index.md}:82:70 - Unknown word ({SpellingError:gatsbyjs})
{FileName*:C:\src\blog\content\blog\2017\04\checking-sql-agent-job-ownership-with-pester\index.md}:17:40 - Unknown word ({SpellingError:msdb})
'@

We've specified that the FileName appears multiple times with an asterisk (FileName*), which generates a record for every occurrence of FileName and it's associated SpellingError. Passing this template to ConvertFrom-String over our output gives us:

$out | ConvertFrom-String -TemplateContent $template

# FileName      : C:\src\blog\content\blog\2020\12\analyze-your-create-react-app-bundle-size-without-ejecting\index.md
# SpellingError : gatsbyjs
# RunspaceId    : 8bc74548-2f8c-4f2b-b350-5f2d5c2dc033

# FileName      : C:\src\blog\content\blog\2017\04\checking-sql-agent-job-ownership-with-pester\index.md
# SpellingError : msdb
# RunspaceId    : 8bc74548-2f8c-4f2b-b350-5f2d5c2dc033

# ...snip...

All that's left now is to group these by SpellingError, and then include a count of files with that error as well as a raw count. The grouping and counting is straightforward - where it gets interesting is the unique count.

We're adding a custom property on Select-Object by creating a hashtable (which takes a label/expression pair). The group (from Group-Object) contains an array of all the objects in that group, so we can count the unique values of FileName with Get-Unique and Measure-Object. It's a rather busy one-liner.

$out |
  ConvertFrom-String -TemplateContent $template |
  Select-Object SpellingError, FileName |
  Group-Object SpellingError |
  Select-Object Name, @{l='# Files';e={($_.Group.FileName | Get-Unique | Measure-Object).Count}}, Count |
  Sort-Object Count -Descending

The full script

$out = npx cspell content/**/*.md*

$template = @'
{FileName*:C:\src\blog\content\blog\2020\12\analyze-your-create-react-app-bundle-size-without-ejecting\index.md}:82:70 - Unknown word ({SpellingError:gatsbyjs})
{FileName*:C:\src\blog\content\blog\2017\04\checking-sql-agent-job-ownership-with-pester\index.md}:17:40 - Unknown word ({SpellingError:msdb})
'@

$out |
  ConvertFrom-String -TemplateContent $template |
  Select-Object SpellingError, FileName |
  Group-Object SpellingError |
  Select-Object Name, @{l='# Files';e={($_.Group.FileName | Get-Unique | Measure-Object).Count}}, Count |
  Sort-Object Count -Descending

Updating babel-eslint to @babel/eslint-parser for React apps

Timothy Addison — Wed, 31 Mar 2021 00:00:00 +0000

As of March 2020, babel-eslint has been deprecated, and is now @babel/eslint-parser. That doesn't stop it (as of March 2021) being downloaded 6.5 million times per week. You wouldn't know this unless you attempted to add it as a new dependency, in which case npm would tell you:

npm WARN deprecated babel-eslint@10.1.0: babel-eslint is now @babel/eslint-parser. This package will no longer receive updates.

Upgrading is straightforward but I couldn't find any clear guides - so if you want to avoid the trial and error you can follow the below steps (which I've tested on Create React App, Next.js, and Vite apps - which all use Babel under the hood).

As to why babel-eslint has been deprecated? It's documented in the (now archived) babel-eslint repo:

As of the v11.x.x release, babel-eslint now requires Babel as a peer dependency and expects a valid Babel configuration file to exist. This ensures that the same Babel configuration is used during both linting and compilation.

Short version

Remove babel-eslint
Add @babel/eslint-parser @babel/preset-react
- You may also need the peer dependency @babel/core (npm7 installs peer dependencies by default)
Update the parser in your .eslintrc.* file (from babel-eslint to @babel/eslint-parser)
Add the following to the parserOptions configuration in your .eslintrc.* file:

requireConfigFile: false,
babelOptions: {
  presets: ["@babel/preset-react"]
}

Longer version

First, switch out the Babel eslint plugin by removing babel-eslint and installing @babel/eslint-parser:

npm uninstall babel-eslint
npm install @babel/eslint-parser -D

If you're not running npm 7 (Node15) you will also need to add @babel/core, which is a peer dependency of @babel/eslint-parser. From npm 7 peer dependencies are installed by default.

And then update the parser (in your .eslintrc.* file):

- parser: 'babel-eslint',
+ parser: '@babel/eslint-parser',

If you run eslint now you'll now get an error about config files:

 0:0  error  Parsing error: No Babel config file detected for C:\temp\site-test\tailwind.config.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files

To fix this we need to modify parserOptions:

+  parserOptions: {
+    requireConfigFile: false,
+  },

And now if we run eslint again? An error parsing React code (specifically jsx), which helpfully tells us how to fix the issue:

 18:4  error  Parsing error: C:\temp\site-test\src\templates\blog-post.js: Support for the experimental syntax 'jsx' isn't currently enabled (18:5):
  16 |
  17 |   return (
> 18 |     <Layout>
     |     ^

Add @babel/preset-react (https://git.io/JfeDR) to the 'presets' section of your Babel config to enable transformation.
If you want to leave it as-is, add @babel/plugin-syntax-jsx (https://git.io/vb4yA) to the 'plugins' section to enable parsing

So let's install the plugin:

npm install @babel/preset-react -D

And then update our parserOptions to pass this option through to Babel:

parserOptions: {
  requireConfigFile: false,
+  babelOptions: {
+    presets: ["@babel/preset-react"],
+  },
},

And we'll finally be able to run eslint!

Spell checking your Markdown blog posts with cspell

Timothy Addison — Sun, 28 Feb 2021 00:00:00 +0000

I love authoring in Markdown as there is almost no friction - just write. When the time comes to publish though - at minimum you'll want a preview, and a spell check. If you're using VS Code then the code spell checker extension is the fastest way to get started (don't be fooled by the fact it says 'code spell checker' - it'll happily check any file, including Markdown).

But what if you've got a lot of posts you want to check? And what if perhaps you haven't been all that fastidious about spell-checking your posts in the past? That's where I was a few weeks ago, and I was happy to find there's a simple solution that requires nothing more than Node 12...

Checking all files in a folder

To check all files anywhere under the content/posts folder that are either Markdown (.md) or MDX (.mdx) we can use cspell by running:

npx cspell content/posts/**/*.md*

We'll then see every file and any unknown words:

Running this in the VS Code terminal allows you to CTRL-click any of the errors and be taken straight to that word in the file.

This is the same library that powers the code spell checker extension (that's what the c stands for in cspell), which means you'll see the same errors reported on the command line and when viewing the file.

This is a good start, but what if we're not happy with the defaults cspell uses?

Changing the cspell defaults

If you're using the VS Code extension then by default any changes you make to the settings will be reflected in the .vscode/settings.json file - which probably isn't what you want if you plan on doing anything with cspell from the command line.

I suggest you create a cspell.json file in the root of your folder:

{
  "version": "0.1"
}

And now any changes you make in the context of the workspace will be placed in this file, which is also used by default when running cspell from the command line. As an example, if I wanted to ensure a particular repo was always checked using British English (rather than the American English default) I could add the following to the cspell.json file:

{
  "version": "0.1",
  "language": "en-GB"
}

The options available in the file are documented on the cspell GitHub page.

Handling unknown words

After you've worked through actual spelling mistakes, you'll likely be left with a lot of unknown words that need to be added to a dictionary or ignored. There are multiple ways to handle this (cspell allows for adding or ignoring words at the user, workspace, file, or block/line level). The workflow I've settled on is:

Words that are probably correct in any context I'll add to the user dictionary (these end up in the VS Code user settings.json)
Words which are specific to a repo (abbreviations, technical terms) I'll add to the workspace dictionary (these end up in cspell.json)
Words which are only correct in a specific file I'll add to that file
Words which are not correct but should be ignored I'll ignore for that file

My blog posts all contain yaml frontmatter which gives me a great place to place words I want to add/ignore for that file only - this is done with comments and some cspell keywords.

---
title: My first blog post
# cSpell:words tjaddison Addison
# cSpell:ignore COOOOOOL
---

This isn't a great example as I'd probably want my blog name (tjaddison) and my surname (Addison) to be added to the workspace dictionary. Because they're added as words they are eligible to appear as suggestions. If I'd used the word 'COOOOOOL' anywhere in the post I'd probably not want to repeat myself (and really I should be asking myself why I'd used it in the first place).

When working on any repos with other collaborators I generally try to avoid adding words to the user dictionary - otherwise those words will show up as unknown for anyone else running a spell check on that repo.

Installing cspell as a dev dependency

If you want to shave a few seconds off your npx cspell command and your repo happens to be a Node.js project then you can add cspell as a dev dependency:

npm install cspell --D

This allows you to add it as a script in your package.json:

"scripts": {
  "spell": "cspell content/blog/**/*.md"
}

Next steps

I've only scratched the surface on what is possible with the cspell package and the VS Code extension - the documentation for the package is worth reading through to get an idea of what can be accomplished. There are a handful of custom dictionaries available, and once you've got your unknown count down to zero it might be worth adding spell checking to your build.

I'm still working on getting my unknown count down to zero, and along the way I've fixed some really terrible typos. Sorry if you've had the misfortune of encountering any of those - at least going forwards I've got a handle on them.

Migrating from the official Firestore SDK to Firebase-Firestore-Lite

Timothy Addison — Sun, 31 Jan 2021 00:00:00 +0000

The official Firestore SDK for JavaScript is pretty big - and if you want to use auth too then you're easily looking at 130KB of gzip compressed code. The size of the bundles is a known issue, and there is even a firebase alpha SDK on the horizon that is set to cut the size by up to 80%.

But if you don't want to wait for the alpha to turn into a production release there is an unofficial alternative in the firebase-firestore-lite package. I've migrated apps and seen 90% size reduction with no loss in functionality (although I'm using basic CRUD only - no realtime or offline support). If you'd like more detail about the performance benefits of migrating see the benchmarks.

The rest of this post will cover what migration looks for Firestore with Google auth.

This is not a drop-in replacement - the alternative SDK has a different (simpler!) API and doesn't have feature parity with the official SDK. For details on what features are missing see what am i giving up by using this.

Updating Packages

In order to use Firestore with Google auth we need to use both the firebase-auth-lite and firebase-firestore-lite packages. We can also remove the firebase package:

yarn remove firebase

yarn add firebase-auth-lite firebase-firestore-lite

Configuring Auth

Rather than needing an instance of firebase (probably instantiated with firebase.initializeApp({...}) you pass your API key directly to auth. You can delete any code using firebase/app.

The below example is all you need to configure Google sign-in, assuming your project had already taken the steps necessary to enable it.

The below code uses React. If you're using something else the main changes will be in how you handle the callback auth.listen to store the signed-in user (or null if the user signed out).

// auth.js
import Auth from "firebase-auth-lite";

const auth = new Auth({
  apiKey: "YOUR_API_KEY",
  redirectUri: window.location.origin,
});
// I spent a while wondering why nothing was working - you need the below to wire up handling of the redirect after signing in
auth.handleSignInRedirect();

const [user, setUser] = React.useState(null);
// This is the callback where you need to store the user details
auth.listen((user) => {
  setUser(user);
});

const signIn = async () => {
  await auth.signInWithProvider(`google.com`);
};

export { user, signIn };

Migrating code

Creating the database instance is a little different:

- import "firebase/firestore";
- import firebase from "../Firebase/firebase";
+ import Auth from "firebase-auth-lite";
+ import Database from "firebase-firestore-lite";

- const db = firebase.firestore();

+ const auth = new Auth({
+   apiKey: "YOUR_API_KEY",
+ })

+ const db = new Database({ projectId: "YOUR_PROJECT_ID", auth })

We no longer have an instantiated firebase instance from firebase/app, so instead we pass through an auth instance with our apiKey. This doesn't need to be the same instance of auth we configured sign-in with. Because we no longer configure firebase (which would have contained our project id) we also need to pass projectId to the Database constructor.

It is possible to re-use the same auth instance from the auth.js example above, though I prefer to not export and expose that to the rest of my app.

Get a single document

A few notable changes:

Collections are referenced by ref rather than collection
- ref actually allows you to refer to documents or collections
Documents in a collection are referenced by child rather than doc
- You can also use a path string (e.g. /collection/documentId)
Documents are returned with their data by default (no need to call data())

- const itemRef = await db.collection("items").doc(id).get();
- const item = itemRef.data();

+ const item = await db.ref("items").child(id).get();
// or
+ const item = await db.ref(`items/${id}`).get();

Get all documents in a collection

The list method is used instead of calling get on a collection. By default the list method only returns a single page of 20 items - so if you want an entire collection you need to pass a large pageSize to the list call.

- const accountsResult = await db.collection("accounts").get();
- const allAccounts = accountsResult.docs.map((d) => d.data());

+ const accountsResult = await db.ref("accounts").list({pageSize: 1000});
+ const allAccounts = accountsResult.documents;

Add, update or delete a document

No changes needed here - the mutate methods all work as they did before (albeit with different patterns to access a collection/individual document).

If using a serverTimestamp (docs) you'll need to import Transform (docs) and use that, rather than the FieldValue.serverTimestamp() from the Firestore SDK:

+ import Transform from "firebase-firestore-lite/dist/Transform"

- const serverTimestamp = firebase.firestore.FieldValue.serverTimestamp();

await itemRef.update({
-  updatedAt: serverTimestamp,
+  updatedAt: new Transform("serverTimestamp"),
})

Query documents

A few differences here:

Rather than chaining calls to where we pass an array of filters (each of which is an array)
We call run on the query instead of get
If we want the document's id we need to query the __meta__ property

-  const itemsResult = await db.collection("items")
-    .where("reportingDate", ">=", fromDate)
-    .where("accountId", "==", accountId)
-    .get();

-  const items = itemsResult.docs.map((d) => {
-    return { ...d.data(), id: d.id };
-  });

+  const itemsQuery = db.ref("items")
+    .query({
+      where: [
+        ['reportingDate', '>=', fromDate],
+        ['accountId', '==', accountId]
+       ]
+    })

+  const itemsResult = await itemsQuery.run();

+  const allItems = allItemsResult.map((d) => {
+    return { ...d, id: d.__meta__.id };
+  });

Summary

If you're looking for a reduced bundle size, easier API, are comfortable with an unofficial SDK, and don't need offline or realtime support - then firebase-firestore-lite is worth checking out. I've only scratched the surface of what is supported - read through the documentation for firebase-firestore-lite for a complete overview.

The alpha SDK sounds promising, though who knows when a production-ready release is coming (weeks, months, years?). Until then (and maybe even after if the API is still so clunky) I'll keep reaching for firebase-firestore-lite.

If you're using Firestore for the first time I'd still suggest you use the official SDK. When you need to search for problems (and you undoubtedly will be - the docs and some of the design choices/behaviors are...interesting - see the database is on fire) you'll need to be searching for the right terms/methods/etc.

Analyze your Create React App bundle size - without ejecting

Timothy Addison — Thu, 31 Dec 2020 00:00:00 +0000

After looking at my [Gatsby site bundle size last month] I figured I should do the same for a few create-react-app (CRA) based sites. If you're wondering why you might want to do that, check out the previous post, otherwise read on for the details.

The following steps have been tested with CRA 4

Step 1 - Dependencies

The plugin we'll use to analyze bundle size is webpack-bundle-analyzer.

To avoid ejecting we'll be using CRACO (Create React App Configuration Override) to let us add webpack plugins to the build.

And finally we'll use cross-env to customize our build scripts in a way that will work on Windows too.

yarn add webpack-bundle-analyzer @craco/craco cross-env -D

Step 2 - Inject bundle analyzer plugin

Create the craco.config.js file in your projects root folder, and paste the below JavaScript in. When we wire CRACO up this will load the analyzer plugin for any production builds. Additionally, it'll default to dumping a json report file in the output folder (useful for CI builds), and only launch the interactive analyzer when the REACT_APP_INTERACTIVE_ANALYZE environment variable is set.

// craco.config.js
const webpack = require("webpack");
const BundleAnalyzerPlugin =
  require("webpack-bundle-analyzer").BundleAnalyzerPlugin;

module.exports = function ({ env }) {
  const isProductionBuild = process.env.NODE_ENV === "production";
  const analyzerMode = process.env.REACT_APP_INTERACTIVE_ANALYZE
    ? "server"
    : "json";

  const plugins = [];

  if (isProductionBuild) {
    plugins.push(new BundleAnalyzerPlugin({ analyzerMode }));
  }

  return {
    webpack: {
      plugins,
    },
  };
};

Step 3 - Wire up CRACO

Modify the package.json file to use CRACO:

"scripts": {
-   "start": "react-scripts start",
+   "start": "craco start",
-   "build": "react-scripts build",
+   "build": "craco build"
-   "test": "react-scripts test",
+   "test": "craco test"
+   "analyze": "cross-env REACT_APP_INTERACTIVE_ANALYZE=1 npm run build"
}

The new analyze command will set the environment variable required to launch the interactive analyzer. We have to prefix our environment variable with REACT_APP_ to have it injected by CRA.

Step 4 - Take a look at your bundles

Your production builds will now have a report.json in the root folder that you can consume from CI builds or look at in production. If you want to use the interactive analyzer you can run the analyze command:

yarn analyze

Monitoring your GatsbyJS bundle size

Timothy Addison — Mon, 30 Nov 2020 00:00:00 +0000

Gatsby will do a great job of serving up a well optimized site - but it is constrained by what you told it to serve. If keeping your site lean is important to you, then looking at your bundle size (and what's in that bundle) before and after adding any new dependencies can be illuminating. If you're currently happy with performance (hundreds dof kilobytes of JavaScript can be added and you'll still score a perfect 100 on a web.dev test) then I highly recommend Dan Luu's web bloat. After you've read that and now feel duly chastened about the state of the modern web, let's look at two ways to analyze your Gatsby site's bundle - interactive, and build-time reports.

Configuring the webpack size plugin

There are a couple of Gatsby plugins which will add webpack-bundle-analyzer for you, though I've opted to consume that package directly.

Gatsby plugins that are wrappers for other packages can be really hit or miss. Will they expose all the features you want, and be updated in a timely manner? In many cases I've opted to consume the packages directly and skip the Gatsby plugin as the amount of code needed is small and I appreciate the opportunity to learn more about the bundler and the analyzer plugin. If you prefer to stick with Gatsby plugins then I'd suggest gatsby-plugin-webpack-bundle-analyser-v2.

First of all you'll need to install the plugin (I'm using yarn in my examples):

yarn add webpack-bundle-analyzer -D

Next you need to add the following to gatsby-node.js:

const BundleAnalyzerPlugin =
  require("webpack-bundle-analyzer").BundleAnalyzerPlugin;

exports.onCreateWebpackConfig = ({ stage, actions }) => {
  const analyzerMode = process.env.INTERACTIVE_ANALYZE ? "server" : "json";

  if (stage === "build-javascript") {
    actions.setWebpackConfig({
      plugins: [
        new BundleAnalyzerPlugin({
          analyzerMode,
          reportFileName: `./__build/bundlereport.json`,
        }),
      ],
    });
  }
};

The docs for onCreateWebpackConfig explain the different stages, and we only want to run the analyzer plugin when we're building our production JavaScript bundles (build-javascript) - this stage only runs during a gatsby build, and not during development.

The arguments we pass through are explained in the below sections and determine if we want to look at our bundle size interactively, or in a JSON report.

Viewing bundle sizes interactively

To explore the current state of your bundle I'd suggest running the analyzer in interactive mode. This will launch a server (defaulting to localhost:8888) that hosts an interactive treemap of your bundles:

To launch the analyzer in interactive mode we need to configure an environment variable (INTERACTIVE_ANALYZE). To support Windows we'll first add the cross-env package:

yarn add cross-env -D

And we'll then add a new script that will launch the build process with this environment variable.

/* package.json */
scripts: {
  "analyze": "cross-env INTERACTIVE_ANALYZE=1 npm run build"
}

Once you run the analyze script the analyzer page will launch in a browser when the build process completes.

Something to keep in mind is that all of these bundles aren't loaded for every page/every browser. A good example is the polyfills.js bundle which won't be loaded on older browsers at all.

Reporting on bundle sizes at build time

By default every gatsby build will log the bundle stats to the file __build/bundlereport.json. You can see the current bundle stats for tjaddison.com.

If you don't want to upload your bundle report to your production site you can either gate the stats behind another environment variable (which you won't set for publishing).

Where this really shines is comparing releases - I use Netlify to deploy my site and take advantage of deploy previews to preview every pull request as a branch. This publishes my branch (including my bundlereport.json), which allows me to take those files and compare the bundle sizes of the branch compared to production.

As I make so few changes to the bundles right now I'm eyeballing the numbers - by using the below code I can print out the stats from production and a branch (in this case I'm comparing the before/after numbers from updating to tailwindcss 2.0 and bumping a few other dependencies):

const logParsedReport = (report) => {
  let parsedTotal = 0;
  let gzipTotal = 0;

  for (const file of report) {
    const { label, parsedSize, gzipSize } = file;
    parsedTotal += parsedSize;
    gzipTotal += gzipSize;
    console.log(`Name: ${label} - ${parsedSize} parsed (${gzipSize} gzip)`);
  }

  console.log("---------");
  console.log(`Total ${parsedTotal} (${gzipTotal} gzip)`);
};

const prod = "https://tjaddison.com/__build/bundlereport.json";
const branch =
  "https://5fc8401d8292a600084390dd--fervent-lovelace-f75e7a.netlify.app/__build/bundlereport.json";

fetch(prod).then((res) => {
  res.json().then((json) => {
    logParsedReport(json);
  });
});
fetch(branch).then((res) => {
  res.json().then((json) => {
    logParsedReport(json);
  });
});

This gives the following output (which I've truncated for brevity):

Name: 942fc3de6d17fbede2785a28a3625a2fd54381eb-469888aa0543c06173b4.js - 18354 parsed (6526 gzip)
Name: app-5ff60cbc19fc47db0c13.js - 66067 parsed (18874 gzip)
Name: component---cache-caches-gatsby-plugin-offline-app-shell-js-16703ee5599528db9f93.js - 499 parsed (351 gzip)
...
---------
Total 341617 (111583 gzip)

Name: 942fc3de6d17fbede2785a28a3625a2fd54381eb-0efa3ed3379b611c2795.js - 18152 parsed (6444 gzip)
Name: app-fd4d3ac6b7f96cbbf5f8.js - 74754 parsed (22347 gzip)
Name: component---cache-caches-gatsby-plugin-offline-app-shell-js-16703ee5599528db9f93.js - 499 parsed (351 gzip)
...
---------
Total 349385 (114705 gzip)

Happily this change has reduced the total bundle size - though as the code isn't actually doing a diff I couldn't tell you where. If you come up with a better piece of diffing code for the output please let me know!

Interesting Links - October 2020

Timothy Addison — Sat, 31 Oct 2020 00:00:00 +0000

Keavy McMinn shares some great examples of what senior technical leadership can look like (all engineering career ladders should not lead to management!).
In addition to being a pretty terrifying read from a security perspective, this fascinating journey through the process of discovering a vulnerability in TeamViewer offers a glimpse at a pretty exciting red-team engagement.
7 helpful design tips that should be useful even if you profess to have no design talent (raises hand)
Ever wondered why Mozilla shows up in so many user agents (even when you're not using Firefox)? - the history of the browser user-agent string will clue you in.
If you enjoyed basecs, you'll probably enjoy it's distributed systems cousin baseds.
An incredible introduction to networking for programmers from routing down to the physical (analog!) layer.
Unlike in RPGs, side quests in reality are pretty key to advancing the 'main story' (getting stuff done).
Rick Branson argues that you you shouldn't count production incidents if you want to improve quality.
When someone tells you they're not biased, perhaps you could suggest this thorough article that reminds us we still have a lot of work to do to remove gender bias in tech.
- And once you get past that lack of bias, maybe have a discussion on meritocracy (hint: it's a a fallacy).
Corp.com is up for sale, and that's a bit of a problem if your Active Directory domain is configured in a certain way (a way which corresponds to a set of defaults proposed by Microsoft in the past!).
In addition to being an (extremely) fast JavaScript bundler, esbuild has a fantastic set of architecture docs that (for a relatively short document) gave me a clear picture of how a bundler works.
Maggie Appleton's site is a growing collection of illustrated essays as well as a digital garden that serve as both great introduction to topics, works of art, and novel metaphors that may help concepts stick (from React hooks to databases).
If you've ever had mixed feelings about Object/Relational-Mapping then you'll enjoy the essay The Vietnam of computer science. Just don't expect a clear resolution at the end.
Trying to understand how accelerated database recovery works is far easier once you've read this article from Forrest McDaniel. The animations bring the processes to life in a way no amount of testing (or re-reading whitepapers) could.
And finally, three great papers covered by the the morning paper:
- Meaningful Availability introduces windowed user-uptime as a better way of measuring service availability.
- Extending relational query processing with ML inference is all embedding ML in a database engine. Using SQL Server cores to run ML sounds pretty expensive, but if it can be fast enough to offset the need to ship the data around, maybe moving the model to the data is the way to go.
- When you might be rolling out 600 releases a day, how do you spot the bad ones? Gandalf is Azure's answer, and it does a pretty impressive job.

You don't have to use graphql to query your Gatsby site's metadata

Timothy Addison — Wed, 30 Sep 2020 00:00:00 +0000

The Challenge

The learning curve for GatsbyJS can be pretty rough. For me, a big part of that was feeling that you had to use GraphQL or you were being distinctly 'un-Gatsby-like'. In addition to the documentation featuring it almost exclusively, every starter (from the official starters to popular community ones) had GraphQL everywhere. Because there were no counter-examples I happily went along and dutifully queried all my siteMetadata via GraphQL, as the docs requested.

For an imaginary component that renders a link to my Twitter bio that would mean the following in gatsby-config.js:

site: {
  siteMetadata: {
    social: {
      twitter: "tjaddison"
    }
  },
}

And then this code to query that value:

// TwitterLink.js
import React from "react";
import { useStaticQuery, graphql } from "gatsby";

const TwitterLink = (props) => {
  const { site } = useStaticQuery(
    graphql`
      query {
        site {
          siteMetadata {
            social {
              twitter
            }
          }
        }
      }
    `
  );

  return (
    <a href={`https://twitter.com/${site.siteMetadata.social.twitter}`}>
      Twitter
    </a>
  );
};

After teaching Gatsby to a few people (and having to apologize for the amount of boilerplate almost every time) and building out a dozen or so sites...I'm pretty confident that there are several better ways of handling this.

Solution 1 - Hardcode

Count how many times you reference the piece of data. Is it one? What would it look like if you hardcoded that value where you needed it?

// TwitterLink.js
import React from "react";

const TwitterLink = (props) => {
  return <a href="https://twitter.com/tjaddison">Twitter</a>;
};

Even if you reference your Twitter bio in three places on your site, maybe it's also fine to hardcode in those three places? You'll almost certainly not be changing your Twitter handle that often (if ever), and if you do change it a find & replace for the full URI (https://twitter.com/tjaddison) is going to make quick work of it.

If you're not building a site to share as a starter, do yourself a favour and keep it simple.

Solution 2 - Reference a JavaScript object

If you're going to reference the data in a few places and/or it's going to change fairly often, then centralizing the data may make sense. Rather than putting the data into Gatsby's GraphQL layer, we'll export the values from a file (I used site.config.js) and then import that in any file that needs it.

// site.config.js
module.exports = {
  siteMetadata: {
    social: {
      twitter: `tjaddison`,
    },
  },
};

// TwitterLink.js
import React from "react";
import config from "../../site.config";

const TwitterLink = (props) => {
  return (
    <a href={`https://twitter.com/${config.siteMetadata.social.twitter}`}>
      Twitter
    </a>
  );
};

In addition to needing less code to consume the data, this method is easier to debug as we're working with plain old JavaScript (console.log(config) is far easier to work with than debugging the Gatsby GraphQL pipeline).

In the example above I've kept the same nesting (siteMetadata/social/Twitter) though I could have used any structure (e.g. socialLinks/twitterBioFullUri).

Solution 3 - Hybrid

One of the strengths of the Gatsby ecosystem is the number of plugins that just work. Some of them rely on the presence of siteMetadata int order to just work and may break if you no longer include that in your gatsby-config.js (one common plugin that expects siteMetadata is the RSS plugin gatsby-plugin-feed). If you want to keep those plugins working (without needing to reconfigure them) you can re-use your site.config.js data:

// gatsby-config.js
const config = require(`./site.config`)

module.exports = {
  siteMetadata: config.siteMetadata,
  // rest of the file...

As long as you include the fields needed by the plugins (gatsby-plugin-feed needs title, description, and siteUrl) they'll work just fine.

So no more GraphQL?

At least not when there is a better alternative!

You can't get very far with Gatsby without having to learn how to work with GraphQL and Gatsby's implementation. My suggestion is you don't start there though, and for simple pieces of data you use the simplest technique that will work. Changing the site title (or your Twitter bio) shouldn't require people to even see GraphQL, let alone start questioning why it has been used.

I remember searching really hard for the 'why GraphQL for this data' when first learning Gatsby, and the Why Gatsby uses GraphQL page doesn't offer much content. I do remember reading:

GraphQL is certainly not required, but the benefits of adopting GraphQL are significant. GraphQL will simplify the process of building and optimizing your pages, so it’s considered a best practice for structuring and writing Gatsby applications.

Based on that I figured I better just go ahead and learn it. Older (maybe wiser?) me now looks at that document as a bit of marketing copy rather than a great assessment of the pros/cons of using GraphQL vs. other methods to populate data on your site. I'd also question what aspects of building the site it simplifies for most use cases! After teaching Gatsby a few times I'm now convinced the pedagogical value of having people deal with the GraphQL concepts just to get their name on a starter...definitely too much.

Speaking of starters - if the prevalence (nigh-exclusivity) of GraphQL in the docs is the genesis of this issue, then the starter library is what boosted this problem into the stratosphere.

Gatsby starters

The first iteration of my site was a mess (it still is). It worked though, and I could have shared it for others to build on. I would have been sharing cobbled together bits of code with multiple solutions to a single problem (e.g. sourcing data). There were edge cases I'd worked around and not documented, and packages included that were no longer required. As something to learn from it would have been a poor starting point.

And that is basically the experience I had when learning Gatsby - after completing the tutorial I picked a few different starters (either official or popular community submissions) and set about reverse engineering the 'why' of various technical decisions that were present in a 'good' Gatsby site.

This was not particularly helpful 😅.

My advice to people learning Gatsby today is to really vet any starter before committing to build on it (lest they find themselves with a partly TypeScript, partly styled-components, partly inline-CSS...'unique' starter). If something looks overcomplicated (in docs, blog posts, starters) then it probably is, consider how you'd solve the problem without Gatsby and see if that is any easier. I'm hopeful this situation will improve over time, otherwise it'll be hard to even suggest Gatsby for technical blogs (over something like GitHub pages or 11ty).

If you want to explore more reading on the 'Gatsby is overkill' theme, I highly recommend the post by Jared Palmer that takes these ideas a little further (I'd like to point out I still think Gatsby is great - but I can see how it's not always the right fit for everyone).

Updating to Tailwind Typography to style markdown posts

Timothy Addison — Mon, 31 Aug 2020 00:00:00 +0000

Last year I shared some CSS to style your markdown posts with Tailwind CSS. Earlier this year a Typography plugin was released for Tailwind CSS that takes care of styling your vanilla HTML for you (the kind generated by GatsbyJS processing markdown files). This post will walk through the process of migrating from a custom stylesheet to the Typography plugin.

To get started I installed the plugin (see the installation instructions on GitHub):

yarn add @tailwindcss/typography

# or by adding the tailwind UI plugin (which includes tailwind typography)
yarn add @tailwindcss/ui

And add to the plugin array in tailwind.config.js:

  ...
  plugins: [require(`@tailwindcss/typography`)],

  // Or if you use tailwind UI
  plugins: [require(`@tailwindcss/ui`)],
  ...

And then update the class on the element that will contain the rendered markdown:

- <article className="markdown">
+ <article className="prose prose-lg">

And done! There were a few differences between my old stylesheet and the new plugin:

Different default text color (bg-gray-700 instead of bg-gray-800)
Different link styling (gray with underline - far too edgy for me!)
Added quotes around inline code and blockquotes
More spacing around headers and between lines
A lower max-width for the container

I was pleasantly surprised to discover that the site didn't look all that different after switching over - my stylesheet can't have been that bad from a design perspective.

Some of the changes I've decided to stick with - I've adopted bg-gray-700 globally as well as the enhanced spacing. The other items though (link styling, quotes, container width) I wanted to undo some of the additions. Fortunately the documentation on Github made this a mostly painless experience.

Updating container width was really easy to update:

- <article className="prose prose-lg">
+ <article className="prose prose-lg max-w-none">

Changing the default link styling required adding a typography key to the tailwind.config.js file and overriding the default a styling:

typography: (theme) => {
  return {
    default: {
      css: {
        a: {
          color: theme(`colors.blue.600`),
          textDecoration: `none`,
          "&:hover": {
            textDecoration: `underline`,
          },
        },
      },
    },
  };
};

Although it looks like a lot of code most of it is boilerplate, and the actual styling overrides are only a few lines. The lines above gave me back my blue links with that all-important underline-on-hover (which maybe qualifies as retro in 2020?).

The final change was to disable the additional quotes (which use before/after content) - what I used was the following addition to the css key in the config:

// in the css: { ...block ... }
"code::before": false,
"code::after": false,
"blockquote p:first-of-type::before": false,
"blockquote p:last-of-type::after": false,

And we're almost done. The one piece of custom styling I had to preserve was the rule that stops prismjs from causing extremely long pieces of inline code (like this one here) to scroll rather than wrap (now targeting prose, rather than my old markdown class):

.prose :not(pre) > code.language-text {
  white-space: pre-line;
}

Overall I'm really happy to be building on a design foundation that will evolve as Tailwind CSS does. It also allowed me to replace about 100 lines of custom CSS with 20 lines of tailwind overrides.

Using GistPad to manage your GitHub digital gardens

Timothy Addison — Fri, 31 Jul 2020 00:00:00 +0000

I love GistPad - it has redefined how I think about working with a markdown based wiki in GitHub. VSCode is my preferred editor for markdown - adding the Gistpad extension takes it to the next level.

From work journal to digital garden

I previously wrote about keeping a work journal with VSCode, and while it was OK, the friction of git (pull, commit, push) really broke the flow. Once I started exploring other knowledge-management systems (see awesome knowledge management) I realized what I was really after was a digital garden.

The ability to link between notes (pages, files, etc.) is key - and so the manual work log didn't scale very far. Browser-based systems like OneNote and Roam Research are closer but have other limitations (price, not markdown, a lot of ceremony in splitting by personal/work/client, etc.). Foam is an exciting newcomer, but still leaves you to manage the git work yourself.

The digital gardens I have (and continue to have) the most success with are actually in OneNote (once configured it's just so easy to access, share, cross-link). As they were mostly private (family or work related) the lack of an easy way to share publicly wasn't really an issue.

Recently I've wanted to start working with the garage door up, and after randomly seeing a tweet on using GistPad for knowledge bases the last thing holding me back (that damned git workflow!) was no longer an excuse. Thank you to whatever Twitter algorithm put that tweet into my feed!

It's only been a few weeks but I'm totally hooked on GistPad and cannot imagine this workflow without it. Managing repos is only one of many features - and some of them could easily be extensions in their own right (e.g. managing gists, code tours, playgrounds). I'd highly recommend browsing the readme and looking at the demos of what it can do.

If you just want to start managing (or just start!) your digital garden, it's only a few steps...

Managing a repo

Get started by installing the GistPad extension. To configure it you'll need to provide a GitHub personal access token with the repo permission, which you can create on the new token page.

By default you'll see all of your Gists and a Showcase - but what we're after is a repo.

Open the command palette and run GistPad: Manage Repository. This will give you a list of all your repos, as well as the chance to create a public or private repo.

I created a public one and called it garage-wiki, as any repo with wiki in the name will get special wiki treatment from GistPad. One of the most significant changes you'll get by being in wiki mode is that you can use [[double brackets]] to create internal links, and GistPad will also maintain backlinks.

An example of how those backlinks show up is below - you can see that I've linked to my webmentions page from three different daily journal entries.

And now you're done - you can create, edit, and delete files in the repo - and as soon as you hit save the changes will be synced back to GitHub. No add/commit/push required.

If you're into journalling you can click the calendar icon next to your repo and it'll create a new page for the current day.

Right now the extension is getting updates all the time, so I would strongly recommend you browse the relevant section of the readme. The ongoing updates around support for Foam are particularly interesting if you're thinking about building your digital garden/personal knowledge base.

In closing, I 😍 GistPad - make sure you give it a try!

Password protect a Next.js site hosted on Vercel

Timothy Addison — Tue, 30 Jun 2020 00:00:00 +0000

Want to password-protect some or all of your Next.js site, but want to stick with a free-tier hosting plan? This blog post will walk through the steps to add the following lightweight protection to your deployment:

If you'd like to jump straight to the source code there is an example repo on Github which is also deployed to Vercel. See the public page as well as a protected page (the password is letmein).

Why roll our own password protection?

In a few cases I've wanted to protect one or more pages on a site, but haven't had any need (or desire!) to integrate an authentication/authentication framework.

Adding password protection with a hosting provider is possible, but expensive. Surge requires you to be on a $30/month plan, Netlify needs the $45/month plan, and with Vercel it's a $150/month addon to the pro plan (which is already $20/month).

You obviously get a lot more for your money than just password protection, and I'd hope their implementations are more robust than what you'll find here (though I couldn't find any documentation on the password protection implementation).

Some notable issues with what we'll build today are:

No rate limiting
No Captcha
Password stored in cleartext in a cookie
No protection for static files (in public)
Requires opting-in on a per-page, per-api-route basis

I definitely wouldn't put anything sensitive behind this protection - I typically use it to protect early proof of concepts or as a placeholder for authentication.

And finally, implementing this will disable Next's automatic static optimization as every page will need some server-side logic (checking if your password cookie is correct). If you were wondering why there aren't many (any?) hosting providers that support password protection on the free tier, this might help explain why.

Getting started

You can either start from a blank Next app, or use one of the templates below to get up and running with a starter that has Tailwind CSS, ESLint, and Prettier, and optionally TypeScript. Note this post will assume you're using the JavaScript example.

# Vanilla
npx create-next-app next-password-protected

# or Javascript
npx create-next-app next-password-protected --example https://github.com/aedificatorum/next-starters/tree/main/tailwind-js

# or TypeScript
npx create-next-app next-password-protected --example https://github.com/aedificatorum/next-starters/tree/main/tailwind

And then run it:

cd next-password-protect
yarn dev
# browse to http://localhost:3000

Our goals are to take this app and:

Allow anyone to visit the main page (/)
Add a protected page (/protected)
Add a login/logout page (/login)

Creating a protected page

Create a new protected.js in the pages folder, and add the following code:

// /pages/protected.js

import Head from "next/head";

export default function Protected({ hasReadPermission }) {
  if (!hasReadPermission) {
    return <div>Access denied.</div>;
  }

  return (
    <div>
      <Head>
        <title>Protected Page</title>
      </Head>

      <main>I am supposed to be protected.</main>
    </div>
  );
}

If you browse to /protected you'll now get the access denied message.

Injecting the hasReadPermission prop

We're going to handle setting this prop in the _app.js file, which you'll need to add to the pages folder. This leverages the custom app feature of Next, and will run this logic for every page.

// /pages/app.js

import App from "next/app";

function MyApp({ Component, pageProps }) {
  return <Component {...pageProps} />;
}

MyApp.getInitialProps = async (appContext) => {
  const appProps = await App.getInitialProps(appContext);

  if (Math.random() > 0.5) {
    appProps.pageProps.hasReadPermission = true;
  }

  return { ...appProps };
};

export default MyApp;

In our initial implementation we're going to give you a 50% chance of viewing the page on each refresh. Try it a couple of times, and you should get to see both access denied, and the protected content.

Rather than roll the dice on every refresh, we'll now check a cookie on each request. To do so we'll use the universal cookie package, which lets us work with cookies in both node and the browser with one package/API:

yarn add universal-cookie

We'll put our cookie name in a consts.js file (debugging cookie name typos is great fun, but we'll skip that step today):

// /consts.js
export default {
  SiteReadCookie: "src",
};

And now we'll check this cookie to see if the client has the right password, by updating _app.js:

// /pages/app.js

// At the top of _app.js
import Cookies from "universal-cookie";
import consts from "consts";

// In the getInitialProps function, instead of our 'random' protection
const cookies = new Cookies(appContext.ctx.req.headers.cookie);
const password = cookies.get(consts.SiteReadCookie) ?? "";

if (password === "letmein") {
  appProps.pageProps.hasReadPermission = true;
}

My consts file is in the root folder (not the pages folder), though I've got absolute imports configured, so no need to import from ../consts.

If you check the /protected page you'll now see you can't access it any more. If you add a cookie with the right password (letmein) in the dev tools you'll be able to get in again.

Keep in mind we've put the password in the source code, so if your source isn't private this is a bit of security theatre. If the source is going to be public you could use an environment variable instead.

Adding password entry

In this case 'logging in' means setting a cookie to the password value. To do this we'll create a login component, and then replace the 'Access Denied' view on the page with the login component.

To do that first create a login component. Note the below is styled with Tailwind CSS with the tailwind ui package for custom forms:

// /components/login.js

import { useState } from "react";
import Cookies from "universal-cookie";
import consts from "consts";

const Login = ({ redirectPath }) => {
  const [password, setPassword] = useState("");

  return (
    <div className="w-1/3 max-w-sm mx-auto">
      <form>
        <label className="block">
          <span className="text-gray-700">Password</span>
          <input
            type="text"
            className="form-input mt-1 block w-full bg-gray-50"
            placeholder="Your site password"
            value={password}
            onChange={(e) => setPassword(e.target.value)}
          ></input>
        </label>
        <button
          type="submit"
          className="mt-3 bg-green-400 text-white p-2 font-bold rounded hover:bg-green-600"
          onClick={(e) => {
            e.preventDefault();
            const cookies = new Cookies();
            cookies.set(consts.SiteReadCookie, password, {
              path: "/",
            });
            window.location.href = redirectPath ?? "/";
          }}
        >
          Login
        </button>
      </form>
    </div>
  );
};

export default Login;

Our component accepts a single optional prop (redirectPath) which allows us to redirect the client after they set the cookie. Note that we use window.location.href as the redirect must trigger a request to the server, and we don't want Next to attempt to use client-side routing.

We can now add our login component to our protected page, and render the Login component if the client doesn't have permission to see the page. This means they either don't have the cookie, or they have the cookie but the password is wrong.

// /pages/protected.js

// At the top of protected.js
import { useRouter } from "next/router";
import Login from "components/Login";

// Inside our function component
const router = useRouter();

if (!hasReadPermission) {
  return <Login redirectPath={router.asPath} />;
}

If you refresh the page you can now try setting the wrong password, or the right password.

Adding a logout page

Right now the only way to 'logout' is to clear your cookies, and while testing you'll appreciate having a logout button.

To solve this we'll add a login page in the pages folder which will allow people to login and logout:

import Head from "next/head";
import Cookies from "universal-cookie";
import Login from "Components/Login";
import consts from "consts";

export default function LoginPage({ hasReadPermission }) {
  if (hasReadPermission) {
    return (
      <>
        <Head>
          <title>Logout</title>
        </Head>
        <div className="w-1/3 max-w-sm mx-auto">
          <button
            className="mt-3 bg-green-400 text-white p-2 font-bold rounded hover:bg-green-600"
            onClick={(e) => {
              e.preventDefault();
              const cookies = new Cookies();
              cookies.remove(consts.SiteReadCookie, { path: "/" });
              window.location.href = "/login";
            }}
          >
            Logout
          </button>
        </div>
      </>
    );
  }

  return (
    <>
      <Head>
        <title>Login</title>
      </Head>
      <Login redirectPath="/" />
    </>
  );
}

If you browse to /login you'll now be able to login or logout.

Conclusion

You've now got basic password protection functionality deployed. Keep in mind that you must add this protection manually to every page you don't want to be public. If you'd like to protect the whole site without messing around with the prop on each page you could handle that in the _app.js, redirecting to a login endpoint or returning a 403.

Even with blanket protection in your _app.js keep in mind that your API routes will need protecting individually, as _app.js only executes for pages.

Hope this was helpful and/or educational!

Tips for 1 on 1s during COVID-19

Timothy Addison — Sun, 31 May 2020 00:00:00 +0000

We're now at around week 10 of COVID-19 in NYC, and it looks like it'll be a major part of 2020, if not beyond. While the business of business still moves forward, I've found that my 1:1s have been more demanding than ever before. I'm lucky to have a great team who have provided me with plenty of support, and I'd like to share some practices that have guided how I've approached 1:1s.

Most of this advice is generic and applies to all 1:1s, even in non-crisis times.

Be vulnerable

It's vanishingly unlikely none of your team are concerned. It's also unlikely that they're likely to bring anything up if you don't allow them to see your own concerns too.

Being able to share that you are concerned about the future, that you don't have all the answers, and that you are looking for support from your team (in addition to providing it) - it all goes a long way towards creating an environment where people can confide in you.

Be supportive of problems

You've probably heard the phrase 'don't bring me problems, bring me solutions'. It has to be some of the worst advice I've ever heard - I think the more honest form of this advice is 'only bring me trivial problems that you've solved, let the systemic ones fester'.

Concern about the future state of society is about as unsolvable as it gets. If you've not created the space where your team feel that they can share concerns or unsolved problems, then you've got no chance here.

Trying to create some weird divide where personal problems get different treatment to business problems dehumanizes people. Don't do it.

Address privilege

We're sitting having a 1:1 in the middle of a global pandemic that we can barely get our heads around. We're both very privileged to still have a job. It's ok to address that, be thankful, and still raise concerns.

Does it sometimes feel absurd to see what is happening in the world and then bring up an issue with some mundane (but annoying) expense policy? Of course it does! That's ok, and calling it out can help people move past that (or at least acknowledge it).

Create deliberate space to address the unfixable

It's great when you're able to help someone. Maybe you come up with a way to level-up someone's career, or figure out the perfect team to maximize the chance of success on a tough project. Falling into the 'debug/fix it' trap is an easy mistake to make as a manager, and there is a whole lot going on right now that cannot be fixed.

Rather than ignoring that bucket, I suggest you address it.

You probably won’t hear someone say ‘there is an unfixable problem I want to talk about’, but you might hear 'I don’t feel productive and it’s bothering me'. Listen for signs that your team wants to talk, and remind yourself that they (and you) are still grappling with the fact that this isn’t just remote work, this is remote work in a pandemic. It helps to mention that, a lot.

Acknowledging the feelings/reality that things aren't the same as they used to be is a conversation worth having, and you should underline that you don't expect these issues to be fixed/go away. Try: It's ok to struggle with these things, I'm not expecting anything more from you there - you're doing a great job.

Some of us have got 10/15/20+ years of expertise in navigating working in an office, and after 10 weeks (less than 1%!) we're wondering why things are harder than they used to be.

Revise your schedule

If you're not meeting weekly with your team right now, consider upping the frequency. This is probably the most disruptive period of many people's lives, and if you are catching up with them every fortnight, month, or quarter - you owe them more than that.

You can always cut a meeting short, and dial back the schedule if you both agree it's superfluous. It's not always easy to ask your manager for more time, so do your team the favour of offering it first.

Rome wasn't built in a day...

...and if you attempt everything above in a single meeting, it's going to be a really heavy 1:1! The stronger the relationship with your team, the easier navigating these kind of topics will be.

If you've not addressed the pandemic and it's impact yet, your next 1:1 is the perfect time to get started.

Stay safe.

Absolute imports with React

Timothy Addison — Sat, 25 Apr 2020 00:00:00 +0000

I'd like to say that all my time writing relative import statements like this has taught me some valuable lessons:

import Gauge from "../../../components/Gauge";

However, I can safely say that if I could go back in time and send myself a link to this post that all I'd be sacrificing is a bunch of frustration, and instead of the above I'd be able to write:

import Gauge from "components/Gauge";

It also means that VS Code's helpful attempts (which for me have had mixed results) at fixing imports when moving files around becomes a thing of the past.

The good news is that this is a very easy fix, as both create-react-app (CRA) and Next.js (Next) will read from jsconfig.json (if it exists) during the build process.

To learn more about the jsconfig.json file, including other benefits it proves, configuration options, and best practices, check the Visual Studio Code documentation on jsconfig

The file needs to go in the root of your project folder, and the contents vary based on whether you're in a CRA or Next project:

// jsconfig.json for create-react-app
{
  "compilerOptions": {
    "baseUrl": "src"
  },
  "include": ["src"]
}

// jsconfig.json for Next.js
{
  "compilerOptions": {
    "baseUrl": "."
  },
  "exclude": ["node_modules"]
}

The key difference is which folder is the base - for Next it's the root (.) folder, and for CRA it's the src folder.

We also want to ensure we exclude node_modules, which is explicit for Next ("exclude": ["node_modules"]) and implicit for CRA (accomplished by only "include": ["src"]). This does not impact your ability to import and use packages (e.g. import React from "react" still works), but it does stop VS Code to not attempt to index and provide intellisense on your node_modules folder.

So, now you've saved some time (and frustration) by binning relative imports, perhaps it's time to consider type checking your JavaScript? If I've wasted minutes on relative imports, then I think I'm into hours (or days!) lost that "checkJs": true could have saved me.

Getting a list of all performance counters ingested to Log Analytics

Timothy Addison — Tue, 31 Mar 2020 00:00:00 +0000

Log Analytics is a great product - easy to get data ingested, and easy to query it. It's also pretty easy to run up a sizeable bill (with the cost being based on GB ingested). I've found it helpful to regularly review what is being ingested to a workspace, which I typically do with the following query:

Usage
| where TimeGenerated > ago(30d)
| where IsBillable == true
| summarize TotalVolumeGB = sum(Quantity) / 1024 by Solution, DataType
| order by TotalVolumeGB desc

You'll get to know which solutions are data heavy pretty fast (I'm looking at you Wire Data and DNS Analytics). Something I've recently spent some time on is performance counters - after the Perf data type crept to the top of the list, so I drilled into that with the _BilledSize property:

Perf
| where TimeGenerated > ago(30d)
| summarize TotalVolumeGB = sum(_BilledSize) / pow(1024, 3) by ObjectName, CounterName // Computer
| order by TotalVolumeGB desc

A recent review showed a fair amount of performance counter overlap between workspaces - my suspicion was that we were ingesting the same counters in multiple workspaces. The following script dumps all counters ingested to any workspace, along with their target instance and frequency. A reliable way to drive up your ingestion costs are the Process/% Processor Time counter ingested for every instance (*) at a frequency of 10 seconds.

We got rid of that one pretty fast.

Import-Module Az

Connect-AzAccount

$allPerfCounters = @()

$subscriptions = Get-AzSubscription
foreach ($subscription in $subscriptions) {
  $subscription | Set-AzContext

  $workspaces = Get-AzOperationalInsightsWorkspace

  foreach ($workspace in $workspaces) {

    $perfCounters = Get-AzOperationalInsightsDataSource -Kind WindowsPerformanceCounter -Workspace $workspace

    foreach ($counter in $perfCounters) {
      $allPerfCounters += [pscustomobject]@{
        Subscription  = $subscription.Name
        ResourceGroup = $counter.ResourceGroupName
        Workspace     = $counter.WorkspaceName
        ObjectName    = $counter.Properties.ObjectName
        InstanceName  = $counter.Properties.InstanceName
        CounterName   = $counter.Properties.CounterName
        Interval      = $counter.Properties.IntervalSeconds
      }
    }
  }
}
$allPerfCounters | Format-Table

Along with a bit of Excel and the results from the earlier queries, it was easy to work through the list and cut down on ingestion costs.

For more information on understanding costs I've found the following articles helpful:

Swapping implementations at compile time with craco and Create React App

Timothy Addison — Sat, 29 Feb 2020 00:00:00 +0000

The Problem

When developing apps I'll sometimes want to mock a dependency for reasons other than unit testing:

Simulating flaky behaviour - error responses, slow calls, etc.
Rapid prototyping of the API experience (before building the implementation)
Working offline
Deploying demo apps with an ephemeral backend (reload to start again!)

I'm a big fan of create react app (CRA), so as much as possible I try to avoid ejecting - as such the rest of this post explores a solution to keep us within CRA.

This typically starts out with something like:

import { getAllPokemon, getPokemon } from "./api-store";
// } from './memory-store'

Judicious use of toggling the comments definitely gets the job done, though once you go beyond a small handful of files it becomes error prone (releasing dev code to prod...).

To solve the issue of the comments being everywhere we can use an environment variable - though unfortunately we can't use it to dynamically import. The following code is what we're trying to do:

// This is not valid code!
import {
  getAllPokemon,
  getPokemon
}
from process.env.REACT_APP_PROD ?
  './api-store'
  : './memory-store';

The solution I settled on uses craco (Create React App Configuration Override) to let us modify the webpack config without ejecting, and the webpack normal module replacement plugin. Once we're done the code will look like this:

import { getAllPokemon, getPokemon } from "./APP_TARGET-store";

The example app is available on GitHub in the craco swap example repo. This commit contains all the changes needed to implement craco.

From comments to craco

Initial State

The starting state for our migration is our app which contains two backends. The code below shows how our getPokemon call is implemented in the api and in-memory versions:

// api-store.js
const getPokemon = async (name) => {
  const result = await fetch(`https://pokeapi.co/api/v2/pokemon/${name}`);
  const resultJson = await result.json();
  const pokemon = { name: resultJson.name, height: resultJson.height };
  return pokemon;
};

// memory.js
const getPokemon = async (name) => {
  // pokemon is a hard-coded array
  return pokemon.find((p) => name === p.name);
};

// app.js
import { getAllPokemon, getPokemon } from "./api-store";
// } from './memory-store'

For this implementation swap to work the method signatures and return types must be the same. One of my favourite mistakes when mocking an SDK is to forget to make my implementations async.

Add craco

In addition to craco I'm installing cross-env, which allows an environment variable to be set in a way that works over Mac and Windows.

yarn add craco
yarn add cross-env

Once craco has been installed, you need to update your package.json to use the craco command instead of react-scripts. This ensures that any configuration overrides we specify will get picked up.

We're also adding a new start method which will set an environment variable. This is how we'll decide which implementation is built, and I typically leave the default (no variable) to build the production version.

"scripts": {
  "start": "craco start",
  "start:memory": "cross-env REACT_APP_MEMORY_STORE=1 craco start",
  "build": "craco build",
  "test": "craco test",
  "eject": "react-scripts eject"
}

At this point your app should still work - though you need to toggle comments in order to switch implementations. We'll fix that next.

Swap implementations based on an environment variable

To hook into webpack we'll create a craco.config.js file in our project root. We check to see if the environment variable is set, and if so use the memory implementation, otherwise default to api. We then add a plugin and tell it that if it gets any requests to import a module with APP_TARGET in the name (e.g. from an import statement), to replace APP_TARGET with either api or memory, and import that instead.

const webpack = require("webpack");

module.exports = function () {
  const appTarget = process.env.REACT_APP_MEMORY_STORE ? "memory" : "api";

  return {
    webpack: {
      plugins: [
        new webpack.NormalModuleReplacementPlugin(
          /(.*)APP_TARGET-(\.*)/,
          function (resource) {
            resource.request = resource.request.replace(
              /APP_TARGET-/,
              `${appTarget}-`
            );
          }
        ),
      ],
    },
  };
};

We now need to modify our code to request the import as follows:

import { getAllPokemon, getPokemon } from "./APP_TARGET-store";

And with that we're done - a different experience when we yarn start vs. yarn start:memory!

Any changes to craco.config.js will require you to restart the dev server, as the file is only process at build/compile time.

Interesting Links - February 2020

Timothy Addison — Fri, 07 Feb 2020 00:00:00 +0000

A tweet on Criteo's migration to .Net core which has a bunch of interesting comments in the thread. The migration is doing good things for Criteo's web app that currently runs on 5,000 32-logical-core hosts, peaks at 6 million requests per second, and has an median latency of 10ms (-10% in core!).
The JavaScript Pipeline Operator (|>) proposal makes for interesting reading - the competing proposals are in-depth and worth reading. Also great to see some examples of before/after - I prefer proposal 1, but would also be happy if 4 makes it.
There are times in interactions with the Azure (notably the billing department) that I can't help but recall this (AWS variant) image.
The Morning Paper with an excellent summary of the Stella Paper, which seeks to understand how operators handle and avert catastrophe in the face of operating complex systems. On the back of the summary I read the whole paper, and (maybe because I'm familiar with the complexities of operating an ecommerce app?) I'd recommend the summary over the paper.
I've previously spent time talking vaguely about why beyond a certain point focusing on force-multiplier/leveraged work is key, this excellent write-up says it far better than I could. If you've even a passing interest in organizational design this is highly recommended reading.
One way to generate a tonne of leverage across an engineering organization is to align multiple groups on a single Technical Strategy. Implementing that may require generating an Architecture Strategy. Once built the strategy is useless if it isn't shared, which is why I recommend reading the companion post that covers how to create and share architecture initiatives.
We've now been using RFCs in engineering for about a year, and as such have taken some time out to reflect on if the process is delivering the value needed. One thing we recently changed was an explicit readme (answering FAQs that we could have anticipated early on). Re-reading this post from Phil Calçado I can confirm this is extremely helpful advice if you're considering an RFC process.
Expectation vs. Reality - Algorithms in the interview vs. Algorithms on the job. Dan Luu telling it like it is, though it doesn't change the fact that most hiring loops are still expecting you to farm leetcode.
A lot to agree with in this post that argues the only code worth a damn is the code running production. The only point I'd hedge on would be the Buy vs. Build point - where I'm actually closer to Buy & Integrate vs. Build & Operate (even that breaks down once you stop squinting). I'm not sure yet if this is my optimization function being different, location along product-market fit curve, team culture, something else? Anyway - a thought-provoking article!
When thinking about team's, do you first stop to ask question 0 - Is this really a team?
Whatever side of a 1 on 1 you're sitting on, it's always helpful to see how others approach their 1 on 1s.
Zero Trust is dangerously close to becoming a marketer's term first & foremost (maybe it's already too late?). Alex Weinert has a great overview (and a linked talk) that spells out what Zero Trust might look like, and if you're a Microsoft shop there are some explicit next-step videos you can consume (this is Zero Trust, not Zero Marketing). The original Beyond Corp resources from Google are worth reading if you want more.
I'm enjoying Designing Data Intensive Applications - the easiest way to know if this is for you is to check out this summary post, and if you're curious you'll almost certainly enjoy the book. The footnotes on GitHub are also worth browsing.
Will Larson's post on the first 90 days as VPE/CTO is worth reading for the same reason I encourage everyone to read books like The Manager's Path - you'll do whatever it is you do far better when you understand the demands of your role, your peer's roles, and your manager's role. And if you're a budding VPE/CTO, there aren't many articles out there written for you.
Also from Will Larson, Engineering Brand explores the challenges of building your company's engineering brand.
Although dated from late 2018, this post on the history of React-Redux is both interesting, educational, and answers a lot of questions I had about 'why are things this way' when working with Redux.
Finally, a pretty dark framing of the future of the internet, is Internet of Beefs, a piece that argues that the current (and future?) state of the internet is a disinformation warzone (that's my paraphrasing, and doesn't do the article justice). I couldn't help but be reminded of the internet as imagined in Neal Stephenson's Dodge. Maybe we're already there?

Remark Slides Starter

Timothy Addison — Sun, 26 Jan 2020 00:00:00 +0000

After spending so long authoring technical blog posts/documentation in Markdown, it's a jarring transition to get dropped back into PowerPoint or Keynote and produce a useful technical presentation...I found myself pasting a lot of screenshots of code, or fighting with formatting. The good news is there are plenty of libraries out there which allow you to author in Markdown.

I start all my technical presentations with the remark-slides-starter, that is built on Remark, DeckTape, and Live Server. If you want to get started right away:

git clone https://github.com/aedificatorum/remark-slide-starter.git new-slides
cd new-slides
npm install
npm run present

You'll need nodejs installed for this to work. I recommend VS Code as the editor for your Markdown slides.

Otherwise read on for some details of the changes I made, and why I think this is a great default for your presentations.

Author - Remark

Remark is an in-browser slideshow tool that is driven by Markdown, and supports a couple of enhancements to Markdown that fill in a few gaps that are needed for presentations (speaker notes, slide build, formatting and layout). Crucially for me, it also has great syntax highlighting out of the box.

The default remark setup has you paste the contents of your slides into a textarea - this works but it means you'll get no syntax highlighting, and large presentations can get pretty unwieldy.

To combat this I split my slides out into one (or more) Markdown files, and load them in with JavaScript. This has the added benefit of allowing slide re-use (import the fragment to multiple presentations):

<!-- index.html -->
<body>
  <textarea id="source"></textarea>
  <script src="https://remarkjs.com/downloads/remark-latest.min.js"></script>
  <script>
    fetch("/slides.md").then((response) => {
      response.text().then((value) => {
        document.getElementById("source").innerHTML = value;
        window.slideshow = remark.create();
      });
    });
  </script>
</body>

As a result of this split you can use VS Code's preview functionality as you work on the slides:

I'd encourage you to check out the Remark Slideshow as well as the Remark Wiki to get an overview of all the functionality provided.

All styles and fonts overrides are contained in the styles.css file.

Note that there is an incredibly popular library called remark (which lives in the GitHub org remarkJS) - it has nothing to do with the remark library I'm discussing in this post. The fact that the Remark slide library owns the remarkjs.com domain name only adds to the confusion!

Export - DeckTape

DeckTape provides PDF export functionality. In order to use you'll need to be serving your presentation, and then run the export command. DeckTape works by spinning up a headless Chrome instance and 'printing' the slides. The results are great, though it can take a little while if you have a very long presentation (50+ slides).

# In one terminal
npm run present

# In another terminal
npm run export

By default the slides will export to a file named slides.pdf.

Present - Live Server

The starter uses Live Server to run the presentation. This will launch a web server that serves the contents of the public folder, and will be available by default on port 8080. Browse to http://localhost:8080/index.html to see your presentation.

If you need to customise the port, edit the package.json file

The great (greatest?) thing about Live Server is support for hot reload - as you edit any file (CSS, Markdown) the server will reload the file and you'll see your changes right away.

Live Server is super-useful as a standalone tool - if you want to quickly serve any file or folder you can run npx live-server file-or-folder.

Remark has a few tricks up it's sleeve for presentation mode. Try pressing P to enable presenter mode, giving you access to speaker notes, and a view of the next slide. If you press C you'll launch a second window which mirrors the current slide, and allows you to effectively present on a second screen.

I need more!

If Remark isn't enough for you, then there are quite a few options of Markdown-driven-slideshows out there. A good starting point is the list of DeckTape examples - it lists a few frameworks and links to live demos in addition to the exported PDF.

I'm really interested in MDX Deck as it allows for React components to be embedded in the presentation (via MDX - Markdown with JSX).

Adding a new field to a Firestore collection

Timothy Addison — Tue, 31 Dec 2019 00:00:00 +0000

In my app I tried to work exclusively with dates stored in UTC, though found some combination of the Firestore SDK, the browser, and my (lack of) JavaScript skills - round tripping dates was really hard. Saving a date at what I thought was midnight then no longer came back in a query that I, again, thought was from midnight. Date objects are passed on save/query, Firestore actually saves them as a Timestamp, and clear documentation on what should be a pretty common use case for dates - querying on a date range, leads to plenty of questions. I decided to reclaim what was left of my sanity and store unix time instead. I still need to handle local to UTC on the client, though Firestore is no longer doing anything more than storing a number.

After updating the app to handle a new field (dateUnix) going forwards, a backfill of old data was required. The example code below takes every item and stores the converted date value in a new field dateUnix. I was surprised to discover there was no way to query by absence of a field (e.g. .where('date','===', undefined)), and so instead you'll need to loop through every item and update if necessary. The code below is designed to run in the browser and assumes you have your Firebase configuration in a file called firebase.js.

import "firebase/firestore";
import firebase from "./firebase";

const db = firebase.firestore();
const itemsCollection = db.collection("items");

export const bulkUpdate = async () => {
  const limit = 50;
  let allItemsResult = await itemsCollection.limit(limit).get();
  let read = allItemsResult.docs.length;

  while (read > 0) {
    const batch = db.batch();
    let updated = 0;

    allItemsResult.docs.forEach((queryResult) => {
      const doc = queryResult.data();

      if (!doc.dateUnix) {
        updated++;

        batch.update(queryResult.ref, {
          // getTime() returns milliseconds
          // We convert to seconds and remove any fractional part
          dateUnix: (doc.date.toDate().getTime() / 1000) | 0,
        });
      }
    });

    await batch.commit();
    console.log(`Updated ${updated} of ${read} items!`);

    const lastVisible = allItemsResult.docs[read - 1];
    allItemsResult = await itemsCollection
      .startAfter(lastVisible)
      .limit(limit)
      .get();
    read = allItemsResult.docs.length;
  }
};

Some things to note about the script:

We get 50 items at a time
- We could get everything if the collection is smaller, though then the example wouldn't feature pagination
We work in batches of 50 until there are no items left
- Working a record at a time does work, though batching speeds things up
If the document doesn't have a dateUnix property, we add it
We create the new field by using existing data on the document
We use a transaction to
When we get the next set of 50 items, we use the last item we saw to control where to start the next page at
- See Pagination in the Firestore docs
Progress is reported to the console...definitely an MVP implementation!

So far I've only had to do this on collections with a few hundred documents and it finished in less than a second. I also had the luxury of knowing the collection wasn't sustaining any concurrent modifications, so I have no idea how this performs on a busy collection.

One thing I've started to do is add an insertedDate value (using the serverTimestamp) to all documents, which I could have used to order the collection, and only process rows that weren't inserting dateUnix (as I updated the app to load that for all new rows). The other advantage of an insertedDate is that it allows you to keep track of progress or even partition the work (if you've got a very large collection you need to update).

Interesting Links - December 2019

Timothy Addison — Sun, 01 Dec 2019 00:00:00 +0000

This whole paper is galaxy-brain - from the fact that Facebook built a load balancer that leveraged users connectedness to the graph that shows 500 million database queries per second. Full of interesting ideas, and mind-boggling numbers - Taiji: Managing Global User Traffic for Large-Scale Internet Services at the Edge is an excellent read.
The state of distributed tracing in .NET Core 3.0 is looking pretty good, and the move to support the W3C Trace Context spec and OpenTelemetry SDKs makes it easier than ever to instrument a distributed app.
Dating from last May, the paper on Accelerated Networking in Azure is interesting both as a reminder of how hard the hosting providers are working to squeeze every last drop of performance out of their compute, and as another example of how reducing team handoffs (in this case hardware to software) leads for better delivery
GatsbyJS now has support for hot schema rebuilds - I've been caught out by not restarting gatsby develop when updating the schema, so this is a welcome addition. No announcement post yet, but you can see the detail in this pull request.
I love Netlify - never has a platform done exactly as it's promised quite so easily. Turns out there is a whole lot more to Netlify than just hosting though, and the complete intro to Netlify has an excellent set of notes that enumerate the rather impressive set of features.
Now I understand what npx does!
If you're interested in online experimentation then the morning paper's coverage of the PlanAlyzer paper is a great place to start, and I found myself following every link. The PlanOut documentation is fantastic, as is the paper that introduces PlanOut:
- Designing and deploying online field experiments
Another Microcode update that's coming along to indiscriminately cause performance issues. To put this most recent change into context, an excellent paper that covers how syscalls have only gotten worse over time (and you'll note a few significant regressions associated with the Spectre update):
- An analysis of performance evolution of linux's core operations
- Damageboy's thread
Choose boring technology is an evergreen idea, and it's nice to get the occasional story of how choosing not-boring code ended up being a (wait for it) bad idea, and the boring code won out in the end:
- Choose boring technology talk
- Dan McKinley's thread on the python middle-tier at Etsy

Link Roundups

Timothy Addison — Sat, 30 Nov 2019 00:00:00 +0000

I've started to publish link roundups to the site. I used to share these in weekly/fortnightly newsletters at work, and as I invariably re-shared most of them outside of work anyway, figured I'd put them on the blog. I used to do these weekly, though think I'll settled for an irregular cadence that sees them come in once or twice a month.

The links pages are implemented as markdown pages, and in order to prevent them showing up as blog posts (and in all the blog meta pages like tags, history, etc.) I used the gatsby-remark-source-name package. My gatsby-config.js now contains two markdown imports:

{
  resolve: `gatsby-source-filesystem`,
  options: {
    path: `${__dirname}/content/blog`,
    name: `blog`,
  },
},
{
  resolve: `gatsby-source-filesystem`,
  options: {
    path: `${__dirname}/content/links`,
    name: `links`,
  },
}

Anywhere I need to refer to either blog posts or links posts I can use the name attribute to disambiguate the allMarkdownRemark node - the following is an example from the gatsby-node.js file that builds the /links pages:

{
  allMarkdownRemark(
    filter: { fields: { sourceName: { eq: "links" } } }
    sort: { fields: [frontmatter___date], order: DESC }
    limit: 1000
  ) {
    edges {
      node {
        fields {
          slug
        }
      }
    }
  }
}

You can see all the changes needed to add the new post type in this pull request. There are probably a few rough edges - I haven't got that many links pages so pagination and navigation may require some fixes.

Once I've got a few more links pages up I'll see about adding a 'recent updates' component to the home page, which would show a feed of posts (links or blog entries). The RSS feed probably needs some attention to, as does the navigation (now there are blogs and links to worry about!).

For now I'm just glad to share the keystrokes with a wider audience.

Interesting Links - November 2019

Timothy Addison — Fri, 08 Nov 2019 00:00:00 +0000

Task-Based Effectiveness of Basic Visualizations is an excellent paper that compares and ranks different visualizations for various tasks based on accuracy, speed, and user preference. Figure 3 is an excellent summary. There is a summary available at The Morning Paper.
Not only does Tether and Bitfinex make a compelling case for the whole thing being a billion dollar fraud, it also servers as a great introduction to cryptocurrency exchanges.
Healthchecks at scale is a great example of how something trivial in the small gets very complicated, very quickly when the numbers get large.
Once upon a time I thought building technical leverage in an infrastructure team would be easy. Of course people were going to flock to the new, easier, better way of doing X. Experience has since disabused me of this notion, and the article Why New Technology Is A Hard Sell offers four compelling explanations, highlighting that being technically superior isn't enough.
John Cutler is always sharing thought provoking ideas in the product management space. The shifts from waterfall to product team is an interesting framework when thinking through what kinds of handoffs exist, and where.
The React Conf 2019 videos are now available on YouTube
Although targeted more at library authors, the post Building Great User Experiences with Concurrent Mode and Suspense introduces some big ideas (if you want some more concrete check out the React Concurrent docs)
I've been iterating on weekly updates for a while now (this link post represents the most recent change - rather than an internal linkdump they're now public). I always enjoy reading about how other people approach them.
Typescript 3.7 is here, and it's a pretty big release
And finally - when free isn't really free, the latest from system32comics on Instagram.

Adding cover images to your Gatsby blog

Timothy Addison — Sun, 20 Oct 2019 00:00:00 +0000

If you decide to use GatsbyJS for your blog I would highly recommend starting with a simple starter and building up. Gatsby blog starter introduces a handful of concepts and is substantially easier to understand than some of the more advanced starters. One feature that adds some personality to your site is featured images for your posts - both on the post list and on the post itself. The rest of this article walks through the steps needed to go from the Gatsby blog starter to having featured images on the post list and post pages.

Add the metadata to a post

The first thing to do is update our GraphQL schema, which we do by adding a new property to the blog post frontmatter. To ensure our featured image is also accessible we'll specify some alt text. As an example, we'll add the existing image from the 'Hello World' post as a featured image. You can give the property any name you want - I've used featuredimage which contains a src and an alt.

---
title: Hello World
date: "2015-05-01T22:12:03.284Z"
description: "Hello World"
featuredimage:
  src: "./salty_egg.jpg"
  alt: "A salty egg"
---

If you are running gatsby develop you need to restart that task, as schema changes (or more broadly any changes that require re-running gatsby-node.js) aren't hot-reloadable.

You can verify this has worked by browsing the GraphQL schema at http://localhost:8000/___graphql and inspecting the allMarkdownRemark nodes - the frontmatter should now have a featuredimage property. The image below shows that our salty egg post has a featured image, and the other two posts have nothing.

The GatsbyJS image pipeline

Because our starter has the gatsby-transformer-sharp and gatsby-plugin-sharp plugins any images that are found during the build process will be replaced by ImageSharp nodes. The documentation for gatsby-plugin-sharp is worth reviewing, as is the working with images in Gatsby guide. If you want the quick version, where we referred to ./salty_egg.png in our frontmatter, we'll get back an object we can use with gatsby-image - and that means:

Optimized and resized images
Multiple images generated for different breakpoints
Generates placeholder images (for blur-up/traced placeholder effects)

Anywhere you add images to your Gatsby site you'll want to ensure it goes through the appropriate processing to generate the assets needed to use the Img component provided by gatsby-image.

Adding the image to the post list

The post list is rendered from src\pages\index.js, and to add an image to each post we first need to return the image and its alt tag.

We're taking advantage of a query fragment (you can see the source code for the fragments on GitHub), and we're also going to add a few options to limit the maximum size of images that come back:

frontmatter {
  date(formatString: "MMMM DD, YYYY")
  title
  description
  featuredimage {
    src {
      childImageSharp {
        fluid(maxWidth: 1024) {
          ...GatsbyImageSharpFluid
        }
      }
    }
    alt
  }
}

I'm using fluid (which generates images that stretch to fill a fluid container), though we could have also used fixed, which is for images that have (unsurprisingly) a fixed width and height. The gatsby-image docs are a great place to start if you want to dig into the details.

To get the image on our page we now need to update the markup. First up we need to import gatsby-image:

import Img from "gatsby-image";

To add the image to each post we first extract the featuredimage from our node:

posts.map(({ node }) => {
  const title = node.frontmatter.title || node.fields.slug
  const featuredimage = node.frontmatter.featuredimage

And finally render the image if the post has one (in my example I've put it between the title and the post - you'll want to spend a bit of time styling this!):

{
  featuredimage && (
    <Img
      fluid={featuredimage.src.childImageSharp.fluid}
      alt={featuredimage.alt}
    />
  );
}

And there we have it, a rather functional looking image on our post list!

Adding the image to the post page requires repeating the same steps in the \src\templates\blog-post.js file. Depending on your layout you may want to tweak the settings (max size/height - or maybe you'll want to override the quality and have higher quality images on your post page?).

Migrating from Jekyll to Gatsby

Timothy Addison — Mon, 30 Sep 2019 00:00:00 +0000

After some tinkering with GatsbyJS, I was pretty excited about moving my blog over from Jekyll - great performance, SEO and accessibility out of the box were one reason, and the fact I know far more React/JS than Ruby was another. This post covers my experiences, as well as a few shortcuts that helped make the transition relatively painless.

A little history

When I first started the blog I was fairly committed to getting up and running as fast as possible - spend time writing content, not endlessly tinkering with a site bereft of any content. In some previous attempts to start blogging, I'd get derailed when I started to write my own blog engine. As you can imagine those efforts didn't amount to a great deal.

Although I was aiming for low-friction I did still impose a few constraints:

I should own my content (eliminated blogging on some platforms)
The content to be trivially portable (eliminated Wordpress, etc.)
Content should be easy to write anywhere (this led me to markdown)
Low/zero initial and ongoing cost
A template I could live with (desktop + mobile)
No ads
Fast

Jekyll on GitHub pages was what I settled for, largely influenced by this post from Phil Haack.

Fast forward to today and I've been pretty happy with the authoring process. There are a few rough edges, though I blame most of them on my lack of experience with Ruby (rounds down to zero).

The new stack

After building a few other projects with GatsbyJS + Netlify I realized that my current blog workflow needed update. When I ran my blog through Lighthouse I realized that I'd been neglecting visitor experience because I was happy with my authoring experience.

Performance

Moving the blog posts to the Gatsby pipeline had a significant impact on all the Lighthouse metrics (higher is better):

Test	Old	New
Performance	87%	100%
Accessibility	74%	97%
Best Practices	79%	100%
SEO	82%	98%

These scores don't capture the significant impact that Gatsby's image toolchain can have. In keeping with the 'low friction' approach to authoring I had been...a bit lazy with my images, and rarely bothered to compress them or optimize them for the web. Since moving to Gatsby that all comes for free in addition to a bunch of other optimizations - see gatsby-remark-images for more details.

To demonstrate the impact this had, take a look at the impact of running images through the Gatsby pipeline had for the about page (lower is better):

Metric	Without	With
First Contentful Paint	1.1s	0.8s
Time to Interactive	1.8s	1.6s
Max Potential Input Delay	640ms	70ms

It's easy to forget that not everyone is browsing on a 4G connection from a desktop device. Speed matters, and having that baked in for free was great.

Development

One of the most compelling reasons to migrate was moving to a language/stack I was comfortable coding in. Gatsby sits on top of React/JavaScript - I'm no expert there but I'm infinitely more qualified in JavaScript than Ruby. There was a bit of a learning curve with GraphQL (which I'd never used before Gatsby), though the documentation of both GraphQL and Gatsby are excellent, and if you do make the switch I'd encourage you to read and digest them.

Having a little more experience with frontend development I was comfortable dropping the requirement for an 'out of the box template' that I'd be happy with, and I started with nothing more than gatsby new and built from there. I've been very impressed with Tailwind CSS and enjoyed building a template from the ground up and finishing it (take that 2013-era Tim!).

Developing with Gatsby is extremely smooth, and I love the fact I can trivially preview a production version of my site locally - I'll confess with Jekyll and GitHub pages I may have 'committed until it works'. Again - I'm sure some of this is down to my ineptitude with Ruby, though I will say the Gatsby experience is very straightforward - gatsby build && gatsby serve.

Hosting

I've never had any issues with GitHub pages hosting, though now I've seen Netlify I don't know if I could go back! A couple of fantastic features:

Generous free tier (including custom domain + SSL)
Deploy previews (open a pull request and Netlify will build a preview for it)
One-click setup (I connected to GitHub and Netlify figured out the rest)
Server-side redirects (I moved my blog to /blog/ so this was important for me)

If I'm feeling brave I can still push directly to master and know my site will get built and published, so this ticks the low-friction box.

Migrating from Jekyll to GatsbyJS

Building the new site

You can (and I did test) go straight to gatsby-starter-blog. There are a bunch of starters available for a blog (Gatsby's strong ecosystem is another reason I moved to it), though I ended up doing quite a bit of customization. The main changes from the template were:

Add Tailwind CSS and Purge CSS
Style the blog posts with Tailwind utility classes
Add opengraph share images to each post
Add pagination
Add an archive page
Add a tag page
Add a responsive menu

You can see the source (as well as the painful journey of trial and error in the commit history!) on GitHub.

The following posts were incredibly helpful in getting up and running:

Migrating blog posts

The migration was fairly straightforward, and the bulk of the work was moving posts and assets around.

For some reason (lost to me now - I'm sure it felt like a good idea at the time) I had my posts and assets organized like this:

# Post
/_posts/2019/03/19/post-title.md

# Post images
/assets/2019/2019-03-19/picture1.png
/assets/2019/2019-03-19/picture2.png

That has always been a minor source of friction in my workflow (why weren't they together!), and the Gatsby way would look more like the following:

# Post
/content/blog/2019/03/post-title/index.md

# Post images
/content/blog/2019/03/post-title/picture1.png
/content/blog/2019/03/post-title/picture2.png

I'm sure it would be possible to maintain the current folder structure and stitch the data together in code, though I wanted to move a post and all associated assets around by copying a folder if needed, so I used the following PowerShell to bulk-move everything. Note that I created a new repository, rather than attempting to migrate in-place.

I migrated a few posts 'by hand' before running a script, which is what $maxMigrated does.

$sourceLocation = "C:\src\tjaddison.com"
$targetRoot = "C:\src\blog\content\blog"
$posts = Get-ChildItem -Path "$sourceLocation\_posts" -Recurse -File
$maxMigrated = "2019-06-30"

foreach($post in $posts) {
    $date = $post.Name.Substring(0,10)
    if($date -ge $maxMigrated) {
        continue
    }
    $year = $date.Substring(0,4)
    $month = $date.Substring(5,2)

    $targetContainer = "$targetRoot\$year\$month"
    $name = $post.Name.Substring(11, $post.Name.Length - 14).ToLower() # remove .md
    $blogFolder = "$targetContainer\$name"

    # Move the blog post
    New-Item -ItemType Directory -Path $blogFolder
    Move-Item -Path $post.FullName -Destination "$blogFolder\index.md"

    # Find any assets (images, source code, etc.)
    $imagePath = "$sourceLocation\assets\$year\$date"
    if(Test-Path $imagePath) {
        Get-ChildItem -Path $imagePath -Recurse -Force | Copy-Item -Destination $blogFolder
    }
}

Because the images had been moved the references in the markdown posts needed to be updated - in hindsight it is a much saner system to have the images co-located and all references relative. I used the following regex in VSCode to bulk-replace all image references, as well as the shareimage frontmatter.

# Images
(\(/assets.*/(.*)\))
(./$2)

# shareimage
(shareimage: http.*/)(.*)
shareimage: "./$2"

What's next?

One outstanding feature I've not yet solved is comments. A few times over the years I toyed with adding Disqus but the level of stuff it injects into the site (ads, trackers, etc.) is frankly obnoxious. I did use Staticman for a while but found that the spam protection just wasn't strong enough and I experienced a few issues around availability. Tania Rascia has a great post to get a system up and running with Node.js/Express/PostgreSQL - this is a little beyond my target complexity, so I'm already toying the idea of rolling my own system atop Cloud Firestore or Netlify Forms.

I'm still chasing down a few areas where the site is not as accessible as I'd like, and I've got some ideas on what an improved home page might look like...but mostly I'm pretty happy with how it looks and excited to focus on writing post again rather than migrating them.

Let me know if you have any comments or suggestions (by email for now, there is no comment system!)

Styling markdown posts with Tailwind CSS in GatsbyJS

Timothy Addison — Sat, 31 Aug 2019 00:00:00 +0000

I'm slowly working on migrating the blog from Jekyll to GatsbyJS. I've decided to use Tailwind CSS to style the blog, which means that out of the box (once Tailwind's pretty aggressive reset has been applied) all the markdown posts end up being unstyled. In the below example the only styling on any element comes courtesy of the gatsby-remark-prismjs package, which handles code styling.

After adding some styling to the site.css file we're able to get the post looking halfway decent.

I wanted to keep the Tailwind reset applied to the non-markdown parts of the site, so the blog template renders each post into a div with the markdown class, and the site.css is updated to include the below definitions. I'm using Purge CSS to strip unused classes, and because these classes aren't directly used anywhere (e.g. className="...") we need to tell Purge CSS to ignore them.

Watch out for PurgeCSS and Prism! If you don't specify prismjs/ as an ignore, your production builds will be unstyled. PurgeCSS only runs for production builds (gatsby build), so make sure you test the look & feel of your site after doing a production build, or you might be surprised when you deploy...

/* purgecss start ignore */
/* Markdown Styles */
/* Global */
.markdown {
  @apply leading-relaxed text-sm;
}

@screen sm {
  .markdown {
    @apply text-base;
  }
}

@screen lg {
  .markdown {
    @apply text-lg;
  }
}

/* Headers */
.markdown h1,
.markdown h2 {
  @apply text-xl my-6 font-semibold;
}
.markdown h3,
.markdown h4,
.markdown h5,
.markdown h6 {
  @apply text-lg my-3 font-semibold;
}

@screen sm {
  .markdown h1,
  .markdown h2 {
    @apply text-2xl;
  }
  .markdown h3,
  .markdown h4,
  .markdown h5,
  .markdown h6 {
    @apply text-xl;
  }
}

/* Links */
.markdown a {
  @apply text-blue-600;
}
.markdown a:hover {
  @apply underline;
}
/* Paragraph */
.markdown p {
  @apply mb-4;
}
/* Lists */
.markdown ul,
.markdown ol {
  @apply mb-4 ml-8;
}
.markdown li > p,
.markdown li > ul,
.markdown li > ol {
  @apply mb-0;
}
.markdown ol {
  @apply list-decimal;
}
.markdown ul {
  @apply list-disc;
}
/* Blockquotes */
.markdown blockquote {
  @apply p-2 mx-6 bg-gray-100 mb-4 border-l-4 border-gray-400 italic;
}
.markdown blockquote > p {
  @apply mb-0;
}
/* Tables */
.markdown td,
.markdown th {
  @apply px-2 py-1 border border-gray-400;
}
.markdown tr:nth-child(odd) {
  @apply bg-gray-100;
}
.markdown table {
  @apply mb-6;
}

/* Wrap any inline highlights `that are really long`, but don't modify
   the setting for codeblocks (inside ```), which are rendered in as:
   <pre><code>...
*/
.markdown :not(pre) > code.language-text {
  white-space: pre-line;
}
/* purgecss end ignore */

This is a pretty workable solution for most of my posts, though there are still a few areas I'd like to improve (notably padding for images, and the appearance at a few breakpoints isn't ideal).

Allowing only specified users to access Cloud Firestore

Timothy Addison — Wed, 31 Jul 2019 00:00:00 +0000

I've been building a few apps recently that leverage Cloud Firestore for data storage. These are personal apps and don't store anything particularly sensitive, though that is no reason to leave them in the default development configuration that let's anyone read/write everything.

Although in many projects I'm the only user, there are handful of others where a few people are using the app. A fairly flexible configuration approach that I use as my default is to only allow access if the user is in an 'allow list'.

I'll show the steps needed to do this below, the pre-requisites are:

Cloud Firestore enabled for the project
Authentication configured for the project with at least one user authenticated
- Every user you want to grant access will need to authenticate with the project as we're using their firebase User UID, which is unique to each project

Implementing an allow list in Firestore

To make this work we're going to create a security rule which will allow users to read/write any part of the database only if they exist in a specific collection, which we will manually populate.

To get started you'll need the User UID of a user who has previously authenticated with the project. In the example below I'm using testuser.

Create a new collection called allow-users and for the first document specify the User UID as the document ID. No need to add any fields (though I've found adding a friendly name to remember what the UIDs map to is helpful).

Now configure your Firestore security rules. If you're using the Firebase CLI you would deploy these using a firebase.rules file, or you can paste into the console and publish.

// firestore.rules
rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {
    match /{document=**} {
      allow read, write: if isSignedIn() && isAllowedUser();
    }

    function isSignedIn() {
      return request.auth.uid != null;
    }

    function isAllowedUser() {
      return exists(/databases/$(database)/documents/allow-users/$(request.auth.uid));
    }
  }
}

From here you can either add additional users to the allow list, or if it's a personal project you're done!

Some things to keep in mind:

There is no longer any unauthenticated access possible to Firestore - if you need Firestore data to render your app/login this won't work
Everyone who can access the app gets full access to everything - read and write

References and further reading

Create React App, Emotion, and Tailwind CSS starter app

Timothy Addison — Sun, 30 Jun 2019 00:00:00 +0000

Get up and running fast with Create React App (CRA), Emotion, and Tailwind CSS by cloning the cra-tailwind-emotion-starter git repo:

git clone https://github.com/aedificatorum/cra-tailwind-emotion-starter.git my-app
cd my-app
npm install
npm start

Tailwind CSS

Tailwind Config

The only change from the default is to add a custom colour (aptly named customcolor) to the themes. The tailwind config is used by both the static CSS and the tw macro.

/* src/styles/tailwind.js */

module.exports = {
  theme: {
    extend: {
      colors: {
        customcolor: "#fde396",
      },
    },
  },
  variants: {},
  plugins: [],
};

Static CSS

By default only the base styles have been added to the static CSS (skipping components and utilities). Anchor has also been updated to be underlined. Note that the tw macro will inline styles, so there is no need to add the component and utility classes here.

/* src/styles/tailwind.css */

@tailwind base;

a {
  @apply underline;
}

PostCSS is used to transform this to tailwind.out.css, which is imported by index.js. The output file is in .gitignore.

`tw` Macro

The tw macro allows Tailwind to be used directly by a JSS (CSS in JS) library. In this case we're using Emotion - see below for examples.

Emotion

The App.js contains a few different ways you can use Tailwind and Emotion together. The below example is a simplified version that demonstrates the various ways to use Emotion or Emotion + Tailwind, in addition to calling out which imports are required in each scenario.

/* src/App.js */

import tw from "tailwind.macro";
/** @jsx jsx */
import { css, jsx } from "@emotion/core";
import styled from "@emotion/styled";

const Section = styled.div`
  ${tw`bg-red-900 min-h-screen flex flex-col items-center justify-center text-xl text-white`};
`;

function App() {
  return (
    <React.Fragment>
      <Section
        css={{
          backgroundColor: "black",
        }}
      >
        This requires the pragma + jsx import
      </Section>
      <Section
        css={css`
          color: hotpink;
          background-color: white;
        `}
      >
        This requires the css import
      </Section>
      <Section css={tw`bg-teal-300`}>This uses the Tailwind macro (tw)</Section>
      <Section css={tw`bg-customcolor`}>
        And this uses the custom color defined in <pre>tailwind.js</pre>
      </Section>
    </React.Fragment>
  );
}

export default App;

For more information see the docs on Emotion styled components, the Emotion css prop.

Credits

The following posts were helpful in putting this starter app together:

Keeping Application Insights Costs Under Control

Timothy Addison — Fri, 31 May 2019 00:00:00 +0000

Application Insights (now part of Azure Monitor) uses a pay-per-GB-ingested model, and charges $2.30 per-GB once you exceed the monthly free limit of 5GB. It may surprise you (it certainly surprised me!) to see that by default an Application Insights resource doesn't deploy with a daily cap of 0.161GB (5GB/month), but actually deploys with a daily cap of 100GB!

Left unchecked each resource like this could end up costing you a cool $7,118.50 per month.

In order to vet your estate and bring it under control, the PowerShell script below will check every Application Insights resource you have deployed against a limit you set, and optionally reduce anything exceeding that limit to a more reasonable cap.

In the above example I ran the script against a newly deployed resource, configured to reduce anything with a cap greater than 10GB down to 1GB.

Script Details

The script has three different configuration options:

$CAP_CUTOFF - the daily GB cap at which to take action - any resources beyond this will be capped to the $CAP_TO value
$CAP_TO - the daily limit to set for anything exceeding the $CAP_CUTOFF value
$ignoreResources - An array of resources to ignore. These resources will appear in the report, but no action will be taken

If you don't want the script to make any changes to your environment append -WhatIf to the Set-AzApplicationInsightsDailyCap command, or comment it out entirely

Once the script executes you'll be asked to authenticate to Azure, and it will then enumerate all subscriptions and all Application Insights resources in those subscriptions.

Once complete the script will report a list of all actions (either Ignore, Under Cap, or Reduce To X) for each resource it encountered. The example below shows this running against an environment where there is a single resource ignored, and then others are all under the cap - in this case set to 10GB.

The Script

Import-Module Az
Connect-AzAccount

$CAP_CUTOFF = 10 ## Free tier = 0.161 - 5GB/month
$CAP_TO = 1
$ignoreResources = @("Subscription.Resource Group.Resource")

$subscriptions = Get-AzSubscription

$actions = @()
foreach($sub in $subscriptions) {
    $subscriptionName = $sub.Name
    $sub | Set-AzContext

    $appInsightResources = Get-AzApplicationInsights

    $aiResources = @()
    foreach($ai in $appInsightResources) {
        $aiResources += Get-AzApplicationInsights -ResourceGroupName $ai.ResourceGroupName -Name $ai.Name -IncludeDailyCap
    }

    foreach($ai in $aiResources) {
        $identifier = "$($subscriptionName).$($ai.ResourceGroupName).$($ai.Name)"

        if($ai.Cap -gt $CAP_CUTOFF) {
            if($ignoreResources -contains $identifier) {
                $action = "Ignore"
            } else {
                $action = "Reduce to $CAP_TO"
                Set-AzApplicationInsightsDailyCap -ResourceGroupName $ai.ResourceGroupName -Name $ai.Name -DailyCapGB $CAP_TO
            }
        } else {
            $action = "Below $CAP_CUTOFF"
        }

        $actions += [PSCustomObject]@{
            Subscription = $subscriptionName
            ResourceGroup = $ai.ResourceGroupName
            ResourceName = $ai.Name
            Cap = $ai.Cap
            MonthlyCost = (($ai.Cap * 31) - 5) * 2.3
            Action = $action
        }
    }
}

$actions | Sort-Object -Property MonthlyCost -Descending | Format-Table

I hope this script helps you keep your Application Insights spend under control!

Should I Automate It?

Timothy Addison — Tue, 30 Apr 2019 00:00:00 +0000

Whenever you need to do something more than once it's often tempting to invest in the process - either by making it easier to repeat or fully automating it.

This post isn't about convincing you that it's time to automate that thing (if you're not already 'automate by default' go check out XKCD 1205 - once you are fully on the automation train come back here). This post is about giving you another tool to help decide to not automate that thing.

Download the Calculator to follow along!

The weekly health check

Let's imagine that every week you have a few SQL Servers you run a health check on - you grab a bit of data on key metrics, compare to a baseline number, and then send a quick summary to the team highlighting anywhere the servers need some attention this week. Let's say it currently takes you 30 minutes, and you do it every week.

Ignoring vacations that's 1,560 minutes a year - almost 26 hours spent pulling data! And if you imagine this check is something you'll do forever (which we'll say is...5 years) then that comes out to just over 3 weeks of time spent on that task alone (making some assumptions about working weeks). Surely we should get on automating that ASAP?

How long is that automation going to take?

In the prior step we took our requirement, it's duration and frequency, and a time frame - and then used those to come up with an automation budget. Although many projects end up with arbitrary deadlines I've rarely found them to be delivered feature-complete or on-time. For automation I'd suggest by focusing on the question how long will it take to automate?

This is a pretty hard question to answer! Depending on the task, the amount to which you automate, and your expertise at the automation process the answer will vary wildly. Rather than focusing on automating the whole thing it's often better to focus on how much time you can save by automating part of a task, and then compute your Return On Investment (ROI) for that piece of automation.

Let's assume that we'll start by automating the collection of metrics - in our example we were connecting to each server manually to run a query, and we're going to build something to allow us to run our query against multiple servers at the same time. That's a fairly tractable project, we're pretty experienced with it, so we say it'll take 4 hours to implement that, which once every week will save us 5 minutes.

If we plug those into the Calculator:

Time to automate - 240
Time saved by automation - 5
Times per month - 4 (once per week)

It's going to take 1 year before the automation investment pays off.

The ROI column tells you how much time you've saved less how much you spent on automation - less than zero means you're in 'automation debt', and more than zero means your automation is paying itself back.

Estimates are...approximate

Most of us really suck at estimates. I've found it extremely helpful to always look at automation based on a range. The calculator defaults to assuming that the best-case might be up to twice as fast - instead of 240 minutes it'll only take 120. It also defaults to assuming your worst case is up to three times slower - so that 240 minutes becomes 480 minutes. Ouch.

These multipliers (and you can adjust them) are what drive the ROI Fast and ROI Slow columns. You can see that in the best case (we automate faster than we thought) we get payback after only 9 months, whereas in the slow case we're waiting up to 2 years.

The results of this automation show that this automation has a long-term payoff - 1-2 years before we land in the definitely bucket (which is where even the Slow ROI shows a payoff). If the process is likely to stay static and not require changes over the next 2 years then this might be worth automating?

So, should I automate it?

This tool allows you to quickly see how long before your estimate pays off. Work prioritisation is ultimately about saying no to many different projects, and if you find out after plugging your numbers in that the payback is in years then you might be able to say no quite quickly.

It's pretty rare you'll ever make a decision solely based on the numbers here - I've found the following to be useful (though not an exhaustive) set of inputs into deciding if the automation is worth it:

Does it totally automate a specific task, removing the need for any manual intervention? This is slightly more valuable than something that is partially automated.
How static is this process, and how often do we expect it to change?
Are there any easier ways to save the time (stop doing the task, delegate it, etc.)
Does it help learn/share a new skill?
Does it automate something that is particularly error-prone when completed manually (increasing accuracy)?
How easy is it to maintain/debug the automation process?
Does it create some kind of technical leverage (automating data collection from multiple servers would have uses outside of this specific task, etc.)
Is saving that time the highest value-add right now (if you instead spend those 5 minutes doing it manually what else could you be working on)?

And that last one is really key, especially when you get into larger projects that might take days or even weeks. Every automation effort invariably pays off over some time period, but whether you could be doing something even more valuable with your time is where the really hard work of prioritisation comes in. Hopefully the calculator will help you decide that automation project you were considering isn't worth putting into that list of 'things that need to be prioritised', and give you one less thing to worry about.

Understanding space usage in Azure Monitor logs

Timothy Addison — Sun, 31 Mar 2019 00:00:00 +0000

Data ingested to Azure Monitor logs is billed per-Gigabyte ingested. As a workspace will typically grow to have data coming from many different sources and solutions it is helpful to have a set of queries that allow you to quickly drill into where exactly the GBs (or TBs!) of data you have stored comes from.

I've found the below queries very helpful starting points for three main scenarios:

Regular monitoring (once/month) to see how data volumes are trending
Reacting to a monitoring alert based on overall ingestion volumes
Testing out a configuration change/new solution and observing the impact on data ingested

The latter is particularly important before rolling a change to a workspace with long retention - you wouldn't want (hypothetically :)) to accidentally ingest 100GB of IIS logs and then be forced to retain them for 2 years...

Workspace solution/data type usage

This query provides a summary of the top solutions and data types in a single workspace, along with how they contribute to overall usage. If your workspace has a lot of different solutions or data types you'll benefit from changing the values passed to the top-nested operators.

Usage
| where TimeGenerated > startofday(ago(30d)) and TimeGenerated < startofday(now())
| where IsBillable
| top-nested 1 of 'All' by AllData = round(sum(Quantity),0)
, top-nested 2 of Solution with others='Others' by SolutionTotal = round(sum(Quantity), 0)
, top-nested 2 of DataType with others = 'Others' by SolutionDataTotal = round(sum(Quantity), 0)
| where AllData != 0
| extend SolutionPct = round((SolutionDataTotal / SolutionTotal) * 100, 1)
| extend OverallPct = round((SolutionDataTotal / AllData) * 100,1)
| project AllData, SolutionTotal, Solution, DataType, SolutionPct, SolutionDataTotal, OverallPct
| order by OverallPct desc

A couple of example results are below - one for a security focused workspace, and one for a workspace which has a bit more going on.

We can see here that this workspace is dominated by the Security event log - almost 70% of all the data is the SecurityEvent data type in the Security solution. This corresponds to the Windows Security Event Log records that are uploaded.

This workspace demonstrates how the top-nested operator works. Inside of the LogManagement solution we've got 2 distinct data types (Perf and a Custom Log - the one ending in _CL). We've also got the 'Others' record which tells us that the longer tail of data types in that solution might be worth looking at as they comprise nearly 30% of the solution total, and 15% of overall data.

Workspace data over time

If you have multiple workspaces it can be helpful to see how their data trends over time. Extending this query to run over additional workspaces is unfortunately a matter of copy-paste:

workspace("primaryWorkspace").Usage
| where TimeGenerated > startofday(ago(30d)) and TimeGenerated < startofday(now())
| where IsBillable
| extend Workspace = "Primary"
| summarize sum(Quantity) by Solution, bin(TimeGenerated, 1d), DataType, Workspace
| union (
workspace("secondaryWorkspace").Usage
| where TimeGenerated > startofday(ago(30d)) and TimeGenerated < startofday(now())
| where IsBillable
| extend Workspace = "Secondary"
| summarize sum(Quantity) by Solution, bin(TimeGenerated, 1d), DataType, Workspace
)
| render timechart with(title = 'Workspace Data Usage')

In this example the primary workspace is responsible for capturing data which will be retained for a long period of time, and the secondary workspace captures higher volume data. Use the dropdown to change between viewing by Workspace, Solution or Data Type.

Note the usage of startofday(...) when filtering on TimeGenerated. Cutting boundaries at midnight at the start/end removes weird artefacts where the graph looks like it is unusually low due to only part of a day's data being counted.

Drilling into a single data type

At some point a graph will invariably point to something that looks a little weird - an upward trend or maybe a huge spike - and you'll want to understand where that is coming from. The good news is Azure Monitor logs provides a pair of virtual columns _IsBillable and _BilledSize that let you drill in to any data type and find out exactly what it is costing you.

Event
| where TimeGenerated > startofday(ago(30d)) and TimeGenerated < startofday(now())
| where _IsBillable == true
| extend computerName = tolower(tostring(split(Computer, '.')[0]))
| where computerName != ""
| summarize TotalVolumeMB = sum(_BilledSize / 1024 / 1024) by computerName, bin(TimeGenerated, 1d)
| render barchart with(title = 'EventLog by Computer')

In this example assume we discovered the Event data type in a particular workspace had a spike - running the query should tell us quickly if this was limited to a single server or all servers. And as you can see below there were three servers that had a very eventful day a little while back!

Auto scale down all Event Hub namespaces with Azure Functions

Timothy Addison — Thu, 28 Feb 2019 00:00:00 +0000

A little over a year ago I lamented the lack of an auto-deflate feature for Event Hubs, and offered a way to programmatically scale down your namespaces. That solution still works, but requires a redeploy each time you wanted to add a namespace. Today we'll look at an upgraded function app which programmatically discovers and scales-down all Event Hub namespaces it has access to.

With the addition of a PowerShell script to grant the appropriate permissions to all of your namespaces, you can be up and running (or deflating) in a matter of minutes.

Function App

The source code for the function app can be found on GitHub in the ScaleDownEventHubs repo. This is an Azure Functions v2 app which is by default configured to run every six hours. When executed the function will:

Discover all Event Hub namespaces it has access to
Query each namespace to check if auto-inflate is enabled
- If auto-inflate is disabled it skips this namespace (assume that the throughput units should be static)
Query each namespace for a tag called ScaleDownTUs
- If this tag is present the value is used as the target to scale down to, if it is not present then the default is 1
Compare the current capacity (Throughput Units) to the target, and if the capacity is higher than the target then reduce the capacity

In order to function (no pun intended!) the app requires the following three variables to be available - this would typically be in the form of an application setting:

var clientId = config["ClientId"];
var clientSecret = config["ClientSecret"];
var tenantId = config["TenantId"];

TenantId is your Azure Active Directory tenant. The other values come from the service principal you use to run the app, which we'll create in the next section.

Note that the function does quite a lot of logging - if you configure Application Insights for your function app you'll be able to monitor what your function gets up to over time

Service Principal

The script below will create a service principal - this is what we'll be granting permissions to so that it can perform operations on our Event Hub namespaces.

Login-AzAccount

$sp = New-AzADServicePrincipal -DisplayName "EventHubScaler"
$applicationId = $sp.ApplicationId

$BSTR = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($sp.Secret)
$password = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto($BSTR)
[Runtime.InteropServices.Marshal]::ZeroFreeBSTR($BSTR)

Write-Output "ClientId: $applicationId"
Write-Output "ClientSecret: $password"

If you set the ClientId and ClientSecret application settings you'll now be able to run your function app. If you want to test locally you can put these into a file called local.settings.json - an example is included in the function's git repo (local.settings.json.example).

Note when testing locally you'll want to change the cron string of the TimerTrigger to something more frequent than every six hours - using 0 _/1 _ * * will have the function run every minute.

Permissions

By default the new service principal has no permissions. In order to let it do something useful we're going to assign it Contributor permissions on all namespaces we have access to.

The below script looks pretty daunting, though by default it will do nothing as $WhatIf is set to $true - when running the script it will echo what it would do. If there is no work to do it will echo the status of each Event Hub namespace it discovers, as shown in the screenshot below. When you're ready to add permissions set $WhatIf to $false.

This script will only assign the service principal to namespaces which have auto-inflate set to true.

Login-AzAccount

$appName = "EventHubScaler"
$appRole = "Contributor"
$WhatIf = $true # set to false to add role

$applicationId = (Get-AzADServicePrincipal -DisplayName $appName).ApplicationId
$subs = Get-AzSubscription
foreach($sub in $subs) {
    Set-AzContext $sub | Out-Null
    Write-Output "Context set to $($sub.Name)"

    $hubs = Get-AzEventHubNamespace
    foreach($hub in $hubs) {
        $hubName = $hub.Name
        $capacity = $hub.Sku.Capacity
        $autoInflate = $hub.IsAutoInflateEnabled
        $maxCapacity = $hub.MaximumThroughputUnits

        $assignments = Get-AzRoleAssignment -Scope $hub.Id `
            -RoleDefinitionName $appRole `
            -ServicePrincipalName $applicationId

        if($null -eq $assignments) {
            $assignString = ""
        } else {
            $assignString = "[$appName ASSIGNED]"
        }

        if($autoInflate) {
            Write-Output "Namespace:$hubName :: TU:$capacity/$maxCapacity $scaleDownTUs $assignString"

            if($null -eq $assignments) {
                if($WhatIf -eq $false) {
                    Write-Output "Adding $appRole role for $appName on $hubName"
                    New-AzRoleAssignment -Scope $hub.Id `
                        -RoleDefinitionName $appRole `
                        -ApplicationId $applicationId | Out-Null
                } else {
                    Write-Output "[WHATIF] Adding $appRole role for $appName on $hubName"
                }
            }
        } else {
            Write-Output "Namespace:$hubName :: TU:$capacity $assignString"
        }
    }
}

Conclusion

If you deployed Application Insights you'll be able to see what your scaler is up to over time with a query like this:

traces
| where message startswith "Updating"
| parse message with "Updating Namespace:" namespace:string "in RG:" resourcegroup:string "from:" fromTU:int  "to:" toTU:int
| extend Op = "ScaleDown"
| union (
traces
| where message startswith "Namespace:"
| extend Op = ""
| parse message with "Namespace:" namespace:string "in RG:" resourcegroup:string "already at or below target capacity (Current:" fromTU:int  "Target:" toTU:int *
)
| order by timestamp desc
| project timestamp, Op, resourcegroup, namespace, fromTU, toTU

One example result set is shown below - this shows multiple namespaces surviving the scaler unscathed (unscaled?), and a couple which were scaled-in. You can also see that there a few namespaces which don't scale down to 1 - these have ScaleDownTUs set on them.

Overhead of Event Hub outputs with Azure Function Apps

Timothy Addison — Thu, 31 Jan 2019 00:00:00 +0000

In a world where you're billed for what you use, it pays to really understand what exactly it is you are using. The pricing model of the Azure Function Apps consumption plan sounds pretty simple (pay based on execution count and execution time) - though as always the devil is in the detail.

In a recent project where we've migrated a workload from a dedicated (on-prem) server to a function app, someone asked what sounded like a fairly simple question:

Do we pay (on the function app side) for the output to an Event Hub, and if so how much does it cost?

This post explores the answer to that question for a trivial C# function, and provides a few pointers to help get your head around consumption billing.

If you're wondering why we wanted to deconstruct the function's cost when an execution unit costs a mere 16 picodollars (16x10^-12), consider what we saw after our first week at full load:

5 trillion execution units... Interesting!

Execution units vs. what you see on the pricing page is covered later in the post.

Consumption billing

Functions hosted on the consumption plan charge you for:

Every function execution ($0.20 per million execution)
Every gigabyte-second used ($0.000016 per GBs)

If the function runs for one second and uses 1GB of memory, your function has used 1GBs. The smallest quantum is 128MB for memory, and 1 millisecond for duration. Billing rounds up, so a function using 1MB for 10us is billed at 0.128GB x 1ms, a function using 129MB for 1001us is billed at 0.256GB for 2ms, etc.

Execution time is measured from the start of the function to the end of the function - and is not CPU time. If your function does any calls to external systems (like an Event Hub) that is billed. This includes the input/output bindings.

This is one reason why your functions shouldn't call other functions - you'll get billed for the execution time of both! If you need to use this pattern then Durable Functions might be a better fit, though prepare for a whole host of new billing complexities to wrap your head around.

Billing information is provided at the function-app level, and not per-function level.

Azure Monitor doesn't expose the GBs billing metric directly, and instead exposes a metric called Function Execution Units. One of these is equal to 1 megabyte millisecond. Converting from GBs to MBms can be done by dividing through by 1,024,000 (GB -> MB is 1024, s -> ms is 1000).

All of these together tell us that:

Any optimisation that doesn't help us cross a boundary of 128MB won't impact cost
Any optimisation that doesn't help us cross a boundary of 1ms execution time wont' impact cost
Adding an Event Hub output is only 'free' if it takes <1ms and has a memory impact that won't take us over a 128MB memory boundary

For more details on the consumption plan check out the Consumption FAQ. Precise details on how the memory usage is billed is pretty hard to pin down - there are a handful of outstanding issues that may need to be resolved before it's straightforward to examine (see here, here, and here).

Test harness

Our test harness is going to take a stock function (return) and compare it to a function which outputs a message to an Event Hub.

The functions below were deployed as separate function apps (to enable parallel testing and still get the granular cost metrics needed). You can view the sample project on GitHub. The deployment was a straightforward 'Publish' from Visual Studio, with each function getting it's own storage account/app service plan (on the consumption tier), as well as an application insights resource. An Event Hub was created to act as the output binding for the second test.

No-Op Test

[FunctionName("NoOp")]
public static async Task<IActionResult> Run(
    [HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = null)] HttpRequest req,
    ILogger log)
{
    return new OkResult();
}

Event Hub Test

[FunctionName("ToEventHub")]
public static async Task<IActionResult> Run(
    [HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = null)] HttpRequest req,
    ILogger log,
    [EventHub(eventHubName: "fatesteventhub", Connection = "eh-connection")] ICollector<string> outMessages)
{
    outMessages.Add("Hello from the function app");
    return new OkResult();
}

Running the Test

This PowerShell script uses runspaces to execute the tests in parallel. We're interested in executing the function a set number of times and examining cost (as opposed to load testing), and so any delay or 'unfairness' due to the client-driver being slow isn't an issue.

Once the code has finished executing we can then go to Azure Monitor and pull the stats on execution count for both function apps and see what overhead there is.

$BASELINE_URI = "https://fa-eh-test-appinsights.azurewebsites.net/api/noop"
$EH_OUTPUT_URI = "https://fa-eh-test-output.azurewebsites.net/api/toeventhub"
$REPEATS = 10000

$tests = @(
    @{ Name = "Baseline"; Uri = $BASELINE_URI },
    @{ Name = "EH Output"; Uri = $EH_OUTPUT_URI }
)

$scriptblock = {
    Param (
        $test,
        $repeats
    )
    Write-Output "Testing $($test.Name) for $repeats repeats at $([DateTime]::UtcNow)"
    $timings = @()
    for($i = 0; $i -lt $repeats; $i++) {
        $timings += (Measure-Command {
            Invoke-WebRequest -Uri $test.Uri -Method Post | Out-Null
        }).TotalMilliseconds
    }

    $summary = $timings | Measure-Object -Average -Maximum
    Write-Output "Finished $($test.Name) - Average $($summary.Average) - Max $($summary.Maximum) at $([DateTime]::UtcNow)"
}

## Thank you https://blog.netnerds.net/2016/12/runspaces-simplified/ !

$pool = [RunspaceFactory]::CreateRunspacePool(1,2)
$pool.Open()
$runspaces = @()

$tests | ForEach-Object {
    $runspace = [PowerShell]::Create()
    $null = $runspace.AddScript($scriptblock)
    $null = $runspace.AddArgument($_)
    $null = $runspace.AddArgument($repeats)
    $runspace.RunspacePool = $pool
    $runspaces += [PSCustomObject]@{ Pipe = $runspace; Status = $runspace.BeginInvoke() }
}

while ($runspaces.Status -ne $null)
{
    $completed = $runspaces | Where-Object { $_.Status.IsCompleted -eq $true }
    foreach ($runspace in $completed)
    {
        $runspace.Pipe.EndInvoke($runspace.Status)
        $runspace.Status = $null
    }
}

$pool.Close()
$pool.Dispose()

Results

After running the test multiple times the numbers were fairly consistent - the function with an Event Hub output binding cost about 18% more to run than a stock function.

After crunching the raw numbers (which you can export from Azure Monitor to a CSV) we get to:

Function	Execution Units
Stock	17,371
Event Hub Output	20,500

The addition of an Event Hub output was costing 3,129 execution units, or about $0.000000049 per execution (from the function app side - don't forget there's a whole set of billing concerns associated with the hub itself!).

To put that into context here is what that translates to per-week in additional cost for the Event Hub output binding for various requests per second:

RPS	Additional Weekly Cost
100	$2.96
500	$14.78
1000	$29.57

It's worth emphasising that this is additional cost, and that at high RPS the execution count component of cost starts to dominate. For a function being executed at 1000 RPS you'd pay:

$164.15 in GBs for the stock function
$29.58 extra in GBs for the Event Hub output binding
$120.96 for the execution count (1000 RPS is 605 million executions per week)
$314.69 in total

It's safe to say that when looking into cost that the presence of a binding itself is not the thing to worry about. Over time as the hosting environment and functions runtime improve I would expect the cost associated with a stock function to go down. For very fast functions execution count will dominate cost.

Closing Thoughts

Cost-control in the cloud can be very... interesting? Knowing exactly how much you're spending is easy, but knowing what we're spending it on (and if that's the most efficient way to solve the problem)...another matter entirely.

This post has scratched the surface of this topic - probably leaving you with more questions than answers. I haven't even got into considerations such as dedicated vs. consumption (some excellent coverage on the Azure from the trenches blog).

For our 5 trillion execution unit app we came to realise pretty quickly that the real issue is the architecture of our system - changing the client to reduce the number of messages sent (via batching) will deliver fair more value than than chasing down micro-optimisations in the function app. Some napkin math reveals we can probably get a 5-10x reduction in call count, and as we know executions dominate our cost that optimisation is falling straight to the bottom line.

Understanding how the cloud services are built and how you're charged for using them is key in building the mental models required to architect a cost-effective deployment - the savings at scale can be quite significant.

Adding caching to your PowerShell scripts

Timothy Addison — Mon, 31 Dec 2018 00:00:00 +0000

Most of the time when some part of a script takes a long time to run and you want to re-use the result you'll store it in a variable:

$databases = Get-ListOfDatabases # assume this is an expensive/long-running query
Invoke-SomeOperation -Server serverOne -Databases $databases
Invoke-SomeOtherOperation -Server serverOne -Databases $databases

As long as the function you're calling allows you to pass in the 'expensive' argument you're fine - but what about cases when the computation takes place inside the function:

Invoke-SomeOperation -Server serverOne # internally calls Get-ListOfDatabases
Invoke-SomeOtherOperation -Server serverOne # internally calls Get-ListOfDatabases

Sometimes it makes sense to rework the script to accept the argument, though other times it can be cleaner to modify the call site to cache the result, rather than changing every function to accept and potentially pass through that argument.

The rest of this post will go through a specific example that motivated caching, share a generic function that implements scriptblock-based caching, and call out a few and gotchas.

The Problem

The SQLChecks library contains a set of Pester tests that are designed to compare expected configuration against actual configuration. The first step of any per-database check is to query the instance (Get-DatabasesToCheck) and get a list of databases to test. This query can end up being very slow (tens of seconds) when the server has a lot of availability group databases.

The typical invocation looks something like this (simplified for clarity):

foreach ($config in Get-SqlChecksConfigs -Path "...") {
    foreach ($check in Get-SqlChecks $config) {
        Invoke-Pester -Check $check -Config $config # calls Get-DatabasesToCheck
    }
}

This pattern is used to enable test results to be sent to Log Analytics as each test completes, rather than waiting for the whole batch to finish. If we weren't looping through tests each Pester test file would only be called once, and the performance issue wouldn't exist.

For each $config-$check combination we'll end up calling the Get-DatabasesToCheck function. The resulting list of databases shouldn't change between checks, so without caching we can end up spending a lot of time waiting to build the list of databases, scaling linearly as we add more checks/more databases.

The wrapper code to run the tests is fairly generic, and supports running tests against the server, the SQL instance, or databases (and potentially other facets in the future). To keep the wrapper code generic and make things performant, I added caching inside the Get-DatabasesToCheck function.

Adding Caching

I wanted to keep the call site changes small (so that any tests against the function Get-DatabasesToCheck were testing that function, not the caching logic). The changes in the function itself end up being very minimal:

# before
$queryResults = Invoke-Sqlcmd -ServerInstance $serverInstance -query $query

# after
$queryResults = Get-CachedScriptBlockResult -Key $serverInstance -ScriptBlock {
    Invoke-Sqlcmd -ServerInstance $serverInstance -query $query
}

The Get-CachedScriptBlockResult function looks like this:

Function Get-CachedScriptBlockResult {
    [cmdletbinding()]
    Param(
        [Parameter(Mandatory = $true)]
        $Key,

        [Parameter(Mandatory = $true)]
        [ScriptBlock]
        $ScriptBlock
    )

    $CACHE_VARIABLE_NAME = "SQLChecks_Cache"

    if (-not (Get-Variable -Name $CACHE_VARIABLE_NAME -Scope Global -ErrorAction SilentlyContinue)) {
        Set-Variable -Name $CACHE_VARIABLE_NAME -Scope Global -Value @{}
    }

    $cache = Get-Variable -Name $CACHE_VARIABLE_NAME -Scope Global
    if (-not $cache.Value.ContainsKey($Key)) {
        $cachedValue = &$ScriptBlock
        $cache.Value[$Key] = $cachedValue
    }
    else {
        $cachedValue = $cache.Value[$Key]
    }

    $cachedValue
}

Note that a better name for this function might have been GetOrAdd-ScriptBlockResult - though that isn't an approved verb, so I went with 'Get'. Anyone have a better idea?

We use a global variable that holds a hashtable to act as our cache. The cache key is a string, and the value can be any object. The flow of the function is:

Check to see if the global variable exists, if not create it
Check to see if the cache key exists
- If it doesn't exist then store the result of invoking the script block in the key
- If it does exist, return the value in the cache

Aside: Why ScriptBlock vs. Objects?

The reason we use a ScriptBlock rather than an object is to keep the call site as neat as possible. If we'd have elected to make the cache function take objects as a parameter then the interaction would have looked like this:

# example of using an object cache instead
# before
$queryResults = Invoke-Sqlcmd -ServerInstance $serverInstance -query $query

# after
$databases = Get-CachedValue -Key $serverInstance
if(-not $databases) {
    $databases = Invoke-SqlCmd -ServerInstance $serverInstance -Query $query
    Add-CachedValue -Key $serverInstance -Value $databases
}

A tricky edge case here is what if the value cached is supposed to be null? We could change our Get-CachedValue function to return a success flag and instead pass a reference to an object we want our cache to populate. This, combined with the fact I wanted to make the caching as easy to add (and not require any logic changes) meant ScriptBlock was the winner.

Storing null values is why the Get-CachedScriptBlockResult tests for existence by looking at keys, not seeing if the value exists.

Testing the Cache

To ensure the cache worked as expected I added a few tests (with Pester, of course). Although basic, these did catch some edge cases and helped build the list of caveats. The below tests leverage Pester's mocking functionality, which let's us test that the ScriptBlock is invoked only once despite repeated calls.

You can view the full set of tests (which cover basic functionality and null caching) on GitHub.

Function Get-ExpensiveToComputeValue {
    "Non-mocked"
}

Describe "Get-CachedScriptBlockResult" {
    Context "calls script block the correct number of times " {
        BeforeAll {
            Remove-SQLChecksCache
        }

        Mock Get-ExpensiveToComputeValue { return "mocked" }

        Get-CachedScriptBlockResult -Key "test" -ScriptBlock { Get-ExpensiveToComputeValue }

        It "calls the function once to populate the cache" {
            Assert-MockCalled -CommandName Get-ExpensiveToComputeValue -Times 1
        }

        Get-CachedScriptBlockResult -Key "test" -ScriptBlock { Get-ExpensiveToComputeValue }

        It "doesn't call the function when the value is in the cache" {
            Assert-MockCalled -CommandName Get-ExpensiveToComputeValue -Exactly -Times 1
        }
    }
}

Conclusion and Caveats

As with any piece of code the most important thing is correctness (and then somewhere behind that come performance and maintainability). If adding caching breaks correctness, you definitely have a problem! If there is no performance problem then avoid caching, as it will often do more harm than good.

Once you're sure caching is for you then take some time to understand the scope of global variables, and what is sharing your cache variable's name (perhaps avoid calling it something like $cache).

There are a few gotchas when it comes to caching and you'll definitely have issues if you expect caching to work via remoting or in a PowerShell job (serialization of ScriptBlocks won't work). You will also need to take care with any Pester tests you have that might need the cache resetting between examples (you'll notice above the call to Remove-SQLChecksCache - this clears the global variable).

With all that said - if you do have a good fit for caching the results can be spectacular. After implementing the cache our daily SQLChecks run came down from 20 minutes to less than 60 seconds. That benefit only grows as more tests are added, and so judiciously applied this technique can be a boon.

One ancillary benefit you get is the ability to inspect certain state (the cache is just a variable, take a look inside and see what your scripts are up to!).

SQL Managed Backups and Operating System Error 87

Timothy Addison — Fri, 16 Nov 2018 00:00:00 +0000

We use SQL Managed backups for our on-premises SQL Servers, and have been very impressed with it (from a speed, management, and cost perspective). Shortly after deploying the solution though the SQL error logs started to log errors when attempting to read managed backups:

BackupIoRequest::ReportIoError: read failure on backup device
'https://allthebackups.blob.core.windows.net/ServerOne/LongFileName.log'.
Operating system error 87(The parameter is incorrect.).

Nothing suggested there were any issues - backups were still being taken, our backup chain wasn't broken (restores were fine) - but this error was being logged all the time.

Understanding where the error came from and how to fix it required a better understanding of exactly how managed backup works.

How does managed backup know what backups are available?

The backup in question was from an availability group - the replica the backup runs from frequently changes, but managed backup (via msdb) always has a full list of available backups for the database.

This doesn't appear to be documented anywhere, but we were able (through a combination of reasoning and monitoring) to establish the following set of operations happens to maintain the list of 'what backups exist for this database':

Managed backup gets a list of files in the target container
Managed backup performs a restore headeronly on each file, to get metadata about that file
Managed backup uses this information to determine what it needs to do (delete old backups, create a new full backup, create a transaction log backup)

Why do we get operating system error 87?

After downloading the file that was generating the error and attempting to restore it, we realised the backup was corrupt. The file size tipped us off to the fact this was probably a partially complete backup (kilobytes rather than megabytes). We were able to generate our own 'corrupt' backups by killing an in-flight backup operation, which generated a partial (and corrupt) backup in blob storage.

Although managed backup will delete backups that are outside of the retention period, in the case of a corrupt backup it has no idea what database it belongs to, nor how old it is. As such it won't delete the file, and it'll sit there in the blob storage container forever.

The key thing we came to understand is that there is no central on-premises list of 'what backups I have taken', and that each instance/replica is responsible for interrogating Azure storage to establish what files exist.

This behaviour is actually a pretty neat feature - unlike relying on information in msdb for prior backups (which could have been deleted, corrupted, etc.) managed backup actually interrogates the backup target to determine what is there.

How do I fix it?

The fix is to delete the corrupted backup file from Azure storage. We currently do this manually (as it happens so rarely) but it could be automated to react to the message in the SQL error log. In the future something like automated storage lifecycle management would allow us to set a policy to automatically delete blobs that exceed our maximum retention period, meaning this error would auto-heal after the retention period.

The root cause of the corrupt backup for us was a (planned!) failover. It's also feasible a network blip could terminate an in-progress backup, or perhaps good old-fashioned storage corruption (which is something we really would care about, especially if every backup became corrupt).

Outside of some rather sparse documentation (if we'd known how managed backup worked we'd have figured this out a lot faster) managed backup is something we've been really impressed with - so if you're hitting this scenario you now know how to fix it.

A big thanks to my colleague Jose for getting to the bottom of this one

Resumable Online Index Rebuilds - SQL 2017's Silver Bullet

Timothy Addison — Sat, 03 Nov 2018 00:00:00 +0000

Every new version of SQL Server comes with a whole grab-bag of new features that open up exciting possibilities, or obviate the need for workarounds (or dirty hacks). In the run-up to deploying SQL 2017 I thought that automatic plan correction was going to be the closest thing to a silver bullet I'd seen in any release so far (in terms of value added vs. effort to implement), but it has been eclipsed for us by the awesomeness of Resumable Online Index Rebuilds (ROIR).

In this post I'll talk through a few of the scenarios where this feature really shines, and how it has transformed the way we think about index maintenance. If you'd like more details about how ROIR is implemented I'd encourage you to read through the excellent paper detailing ROIR - this covers how the online index rebuild algorithm was updated, and also demonstrates how in most cases ROIR outperforms the non-resumable version in terms of performance.

When index rebuilds are kind of a big deal

A few conditions (either individually or in combination) can make index rebuilds a pretty big deal in an environment:

24/7 operations (no such thing as a 'quiet time' or maintenance window)
Index is very large (takes a long time to rebuild)
Transaction volume is high (log truncation being blocked means log utilisation grows rapidly)
Presence of Availability Group (AG) replicas (index build has to be replayed on 1...many servers, potentially getting blocked behind queries)

Having enterprise edition (online rebuilds) and fast local-attached drives (NVMe and availability groups) help with the 24/7 and speed issues, but transaction log volume and AG replicas remain a challenge. In the worst case you end up needing a huge transaction log on your primary replica to support the rebuild itself, and then it can grow even larger if redo gets blocked on the secondary replica.

Those last two bullet points caused us to go through a pretty rigorous scheduling process for any maintenance on indexes larger than 100GB in one of our high-volume databases. What should have been an automated script turned into an affair which needed attention from the DB and App teams, potentially dialling down other operations to reduce load (not to mention the DBA monitoring the log utilisation and invariably sweating bullets as the rebuild chugs along). There may have also been several reported incidents of reporting queries on replicas being killed if they got in front of a SCH-M lock.

Not ideal, and that isn't even the worst of it.

Throwing away good work

On our mission critical instances we have no tolerance for any kind of blocking on critical tables. As such we take an extremely paranoid approach to any index of rebuild:

set lock_timeout 1500;
alter index IX_ImportantIndex on dbo.ImportantTable rebuild with (online = on);

Online index rebuilds aren't entirely lock-free - there is a SCH-M lock taken at the start and end of the process, and although managed lock priority allows you to wait at low priority the options for what to do if your session can't get the lock after the timeout are pretty limited:

Kill yourself
Kill anything blocking you
Elevate to regular lock priority

The second two aren't really options - critical work is either going to be killed, or get blocked (for who knows how long - option 3 could potentially wait forever).

The first option is the only one we'd really entertain, but in our testing we found that on a busy OLTP system waiting at low priority for even five minutes had less chance of getting through than waiting at normal priority for 1.5 seconds. This is obviously specific to our environment, but it is what we ended up going with. This sometimes meant that a 30 minute index rebuild could end up getting rolled back because it couldn't acquire a brief SCH-M lock.

The option of leaving the transaction open and waiting e.g. 1 hour at low priority was problematic due to the log usage. We were never entirely happy this index maintenance setup, and it was relegated to the pile of 'works but not ideal, let's hope we don't have to rebuild critical indexes that often'.

Enter the silver bullet - ROIR

In order to support resumable index operations in SQL 2017 the rebuild algorithm was changed to support batching. Internally SQL Server will operate on batches of rows, committing work every 100k rows and allowing the transaction log to truncate after each batch. In the event that the rebuild is stopped (either by a pause command, or something like a network blip/failover) the maximum amount of work that is 'lost' will be a single batch.

The icing on the cake is that resumable index rebuilds will typically execute faster than the regular, non-resumable variety. With a small change our index maintenance suddenly becomes almost risk-free:

set lock_timeout 1500;
alter index IX_ImportantIndex on dbo.ImportantTable rebuild with (online = on, resumable = on);

If the rebuild gets to 100% complete and attempts to perform the switch (acquiring the SCH-M lock) and fails, we haven't lost all the work. Looking in sys.index_resumable_operations we'll see our rebuild at 100%.

select *
from sys.index_resumable_operations;

The following SQL attempts to complete the rebuild - sitting at 100% the only thing it needs to do is take the schema stability lock to switch out the new index for the old. As we're using a lock timeout of 1500ms it might take a few tries, but crucially we do not have to start the index rebuild from scratch.

alter index IX_ImportantIndex on dbo.ImportantTable resume;

Resumable index rebuild recommendations

First of all - use this everywhere. There are almost no situations in which you don't want to use resumable index rebuilds (this will be easier in SQL 2019 when you can set the options ELEVATE_ONLINE and ELEVATE_RESUMABLE). Even outside of an AG or high-volume scenario, why would you want to block log truncation, or put yourself at risk of losing the rebuild because your network drops (or because SSMS crashed...).

The new risk that resumable rebuilds bring is that you'll end up with a database full of paused rebuilds, all of which are using up additional space and adding overhead to all the DML against their target indexes. I'd recommend at minimum an agent job that looks for any paused indexes which were paused more than N hours ago and aborts them.

In our environment we're currently exploring a solution that gives us much finer grained control over index maintenance by leveraging the MAX_DURATION option. This allows us to tell the rebuild to abort after N minutes, and via an agent job we can optionally choose to resume the rebuild, wait a while, or abort it. We're hoping to use this to better leverage available CPU/IO capacity across the cluster and ramp up/down operations with capacity. Did I mention that you can resume operations with a different MAXDOP?

It's still early days for our experimentation, but the ability to dial index rebuilds up/down depending on how the cluster reacts to the current MAXDOP is almost as awesome as the ROIR feature itself.

Mitigating SERVICE_BROKER_WAITFOR_MANAGER latch waits

Timothy Addison — Mon, 15 Oct 2018 00:00:00 +0000

Our production environment recently started generating alerts on huge blocking chains (100s of requests), which were accompanied by increased database response times for various procedures. The blocking chains all had the latch wait SERVICE_BROKER_WAITFOR_MANAGER in common, some of which were blocking for seconds (adding significant overhead to operations that would normally complete in a few milliseconds).

In this post I'll walk through what an environment leveraging service broker might look like, show you how to reproduce the issue, and offer some mitigation strategies/general advice for service broker at scale.

Service Broker Recap

SQL Service Broker (SSB) can be used as a messaging system to decouple applications. Note that SSB is a complex product and this specific example only covers one way it can be used (or perhaps misused).

Imagine a website application wants to process a card payment - the website would queue a process_payment message (to a SSB queue). That SSB queue is monitored by a separate payment application. The payment application would receive the process_payment message, process the payment, and respond to the website with the result. This is an example of a dialog (two-way) conversation.

SSB also supports fire and forget (care needs to be taken to ensure it is implemented correctly), which you might use if the website wants to send an email, but doesn't need to wait for the result (the website puts an order_confirm_email message on a SSB queue, and the email application processes it from that queue at some point in the future). Background processes can also take advantage of fire and forget to queue up a large amount of work - imagine an email for newsletter_mail for every subscriber.

In a complex application there can be dozens or hundreds of different messages and queues, and multiple applications queueing/dequeueing messaging. Scaling SSB to cope with high throughput is a challenge of it's own (see this whitepaper on scaling SSB from 2009) - the specific scaling challenge I'll discuss in this post relates to waiters (applications waiting to dequeue messages from a queue).

Each queue can have 0 to N requests waiting to process (dequeue) a message. While you could have applications check each queue periodically (polling with receive), SSB allows you to register and 'wait' on a queue, such that the request will block until a message arrives, at which point it will be returned to the calling application. You do this with the waitfor keyword. In our environment we use waitfor to ensure each queue has a minimum of 1 waiter at all times.

The final bit of background needed is that you can use event notifications to have SQL Server notify you when there are messages waiting to be processed on a queue (specifically with the QUEUE_ACTIVATION event). We use this to create additional threads (which call the queue with waitfor) to process multiple messages concurrently.

In summary:

One queue per decoupled process
At least one reader per-queue
The number of readers per-queue will scale as there are more messages to process
The way readers interact with the queue is via the waitfor keyword
This pattern is used for both dialogs (two-way) and fire and forget (one-way) messaging
Dialogs are typically latency sensitive (we care about the response time)
Fire and forget are typically latency insensitive, and can be very high volume

The problem

It starts with alerts for blocking in production. The environment in question normally never has anything but transient (<100ms) blocking, so to see an alert go off with a few seconds of blocking involving hundreds of requests was...disconcerting! The blocking alert helpfully pointed the finger at service broker (SERVICE_BROKER_WAITFOR_MANAGER), though after a quick Google we assumed we were on our own (when the latch waits library says TBD, typically you're in trouble).

In this case however the Microsoft latch docs had us covered. Most of the time 'Internal use only' is all you get, but in this case case the description gave us a few concrete avenues to investigate:

Used to synchronize an instance level map of waiter queues. One queue exists per database ID, Database Version, and Queue ID tuple. Contention on latches of this class can occur when many connections are: In a WAITFOR(RECEIVE) wait state; calling WAITFOR(RECEIVE); exceeding the WAITFOR timeout; receiving a message; committing or rolling back the transaction that contains the WAITFOR(RECEIVE); You can reduce the contention by reducing the number of threads in a WAITFOR(RECEIVE) wait state.

Armed with this, and after checking to see if this was actually leading to increases in response time for latency-sensitive calls (it was), I set out to try and isolate and remedy the waits.

Reproduction

Setup

To reproduce the issue all we need is a single queue, and a way of generating concurrent activity against that queue. Note that This does not constitute what a real deployment of SSB looks like (with multiple queues, request/response services, priorities, etc.).

In order to generate load against the queues I used SQLDriver - you can download the SQLDriver 1.0 standalone binaries, or use any other stress testing tool. Crucially, you want to ensure your tool will tell you about the response times and not just the total runtime.

The database and schema look like this:

create database BrokerTest
go
use BrokerTest
go
alter database current set enable_broker;
create master key encryption by password = 'secure';
go
alter database current set compatibility_level = 140;
alter database current set recovery simple;
alter database current set delayed_durability = forced;
go

create queue dbo.TestQueue;
create queue dbo.TestQueue1;

The reason we set delayed durability to forced is to minimise/eliminate any WRITELOG waits. Although it doesn't appear to be documented anywhere, calling waitfor(receive...) will generate transaction log - even when called on a totally empty queue.

To start with I ran a load test and looked at sp_whoisactive to confirm the wait was showing up. The following command will run 100 concurrent waitfor-receives, each with a timeout of 1ms. Each thread will execute the statement 10000 times.

SqlDriver.exe
  -s "waitfor(receive top(1) * from dbo.TestQueue), timeout 1"
  -t 100
  -r 1000
  -c "server=localhost;initial catalog=BrokerTest;integrated security=SSPI;Application Name='SQLDriver';max pool size=1024"

The numbers that I'll focus on for the rest of the post are the median response time (P50), and the tail latency (P95). In the above example the median response time was 9ms, and the 95th percentile response time was 16ms. In a typical system the timeout would be much higher than 1ms (at such a low timeout we might as well be polling the server), and even with a single thread we'd see variation with timeouts < 10ms - though most of the wait time would be the benign wait BROKER_RECEIVE_WAITFOR.

Eyeballing the waits for this first test I did see the expected latch waits (both LATCH_SH and LATCH_EX for SERVICE_BROKER_WAITFOR_MANAGER), though I also saw the wait type WAITFOR_PER_QUEUE. This wait type is not documented, and seems to indicate there is a maximum number of waiters per queue. In additional testing I got up to 700 concurrent waiters on one queue (before my laptop ran out of worker threads), so this wait appears to be linked with registering a waitfor, rather than the total number of waiters.

Load testing - single queue

To ensure there were no other waits being missed by eyeballing sp_whoisactive, I used an extended event trace for all waits coming from the SQLDriver application.

if exists ( select * from sys.server_event_sessions where name = 'SQLDriverLatches' )
  drop event session SQLDriverLatches on server;
go

create event session SQLDriverLatches on server
add event sqlos.wait_info (
  where sqlserver.client_app_name = N'SQLDriver'
  and   opcode = 1 /* End - otherwise we get the start & end of every wait */
  and   duration > 0
)
add target package0.asynchronous_file_target (
  set filename = N'C:\temp\SQLDriverLatches.xel'
)
with (
  max_dispatch_latency = 1 seconds
);
go

alter event session SQLDriverWaits on server state = start;

/* Run the load test! */

alter event session SQLDriverWaits on server state = stop;

Creation of the extended event session and the analysis heavily borrows from Paul Randall's post on session level wait stats

The results can be analysed with the following SQL - the .xel files should be deleted between each test. Note that this query can take a long time if there are a large number of waits to analyse.

drop table if exists #xetemp;

select cast(event_data as xml) as EventData
into #xetemp
from sys.fn_xe_file_target_read_file ('C:\temp\SQLDriverWaits*.xel',null,NULL,NULL);

select  dat.WaitType
        ,count(*) as WaitCount
        ,sum(dat.Duration) as WaitTime
        ,sum(coalesce(dat.SignalDuration,0)) as SignalWaitTime
from #xetemp as x
cross apply (
  select x.EventData.value('(/event/data[@name=''wait_type'']/text)[1]','varchar(100)') as WaitType
        ,x.EventData.value('(/event/data[@name=''duration''])[1]', 'bigint') as Duration
        ,x.EventData.value('(/event/data[@name=''signal_duration''])[1]', 'bigint') as SignalDuration
) as dat
group by dat.WaitType
order by count(*) desc

The extended event trace only tells us there is a latch wait (LATCH_EX/LATCH_SH), but not the specific latch being waited on. To track that, I cleared the latch stats DMV before each load test, and then grabbed the results after:

dbcc sqlperf('sys.dm_os_latch_stats', clear);

/* Run the load test! */

select * from sys.dm_os_latch_stats
order by wait_time_ms desc;

I ran benchmarks for 1, 10, 100, 200, and 300 threads - each with 1000 repetitions of a waitfor-receive that has a 10 millisecond timeout. In the below table, SWM Average refers to the average amount of time waiting on the SERVICE_BROKER_WAITFOR_MANAGER latch - computed from the DMV as wait_time_ms / waiting_requests_count. Note a thread can acquire latches multiple times during each request, the averages below represent per-latch request attempt.

Threads	50P	95P	SWM Average
1	15ms	15ms	0ms
10	15ms	16ms	0.36ms
100	22ms	33ms	3.59ms
200	32ms	41ms	5.76ms
300	51ms	64ms	9.44ms

Querying for wait type mainly served to confirm the dominant waits are for latches (plus a smattering of the rogue WAITFOR_PER_QUEUE). Examining average latch time shows how this wait type increases along with response time, both median and tail latency.

A high thread count against a single queue will bottleneck on this latch - but what impact does that have if we introduce a second queue?

Load testing - second queue

A common pattern we saw in the blocking alerts was high-volume queues (many messages, and lots of concurrent threads) appeared to be impacting the performance of low-volume queues.

To reproduce this behaviour I started 100 threads with a high repetition count against dbo.TestQueue, and then load tested with low thread count against dbo.TestQueue1 with a second instance of SQLDriver.

For this benchmark we're not able to use the latch stats DMV (as the activity against the first queue will dominate), so we'll look at the response times only.

To capture wait stats related to the second queue only we modify the application name property of the 'background' connection string:

SqlDriver.exe
  -s "waitfor(receive top(1) * from dbo.TestQueue), timeout 10"
  -t 100
  -r 10000000
  -c "server=localhost;initial catalog=BrokerTest;integrated security=SSPI;Application Name='SQLDriverBackground';max pool size=1024"

The requests against the second queue (from a single thread) have a median response time of 15ms, and 95th percentile response time of 27ms. The main waits logged are LATCH_EX and LATCH_SH again, and the latch stats DMV confirms no new latches have shown up.

The second queue is impacted by activity against the first queue - this makes sense as the latch is instance-wide.

Load testing - summary

The latency to acquire the SERVICE_BROKER_WAITFOR_MANAGER latch increases as the number of total concurrent waitfor-receive registrations increases, regardless of the queue.

As an added bonus, there is a mystery wait (WAITFOR_PER_QUEUE) you will run into with a high number of waitfor-receives attempting to register concurrently against a single queue.

So what can we do about it?

Mitigating the problem

The good news is there is a relatively easy workaround - don't wait(for)!

Repeating the previous load test but changing the second queue to be receive (not waitfor-receive) gives a median response time of 0ms, 95th percentile response time of 0ms - and no waits.

This is unlikely to be a change you'll want to make to your application directly, as you'll now be doing 1000 RPS per thread against the server for every queue being received from (assuming a latency of ~1ms).

Instead, we can deploy a receive with a sleep, with the sleep acting as our upper bound for response time. In the worst case you start sleeping right after a message arrives, so you'll wait the sleep duration and then when the next request executes the message will be dequeued.

The SQL for a receive with sleep might look something like this:

receive top(1)
        conversation_handle,
        cast(message_body as xml),
        message_type_name
from  dbo.TestQueue;

if @@rowcount = 0
begin
  waitfor delay '00:00:01';
end;

Other observations

Having hundreds of threads sitting in a waitfor-receive (with high timeout values) had no noticeable impact on new threads registering a waitfor-receive, either on the same or different queues.

The Microsoft latch docs indicate this latch only comes into play when you're modifying the list of waitfors, and so on busy systems where there is lots of concurrent dequeue activity I'd expect similar bottlenecks to manifest (dequeueing a message into an existing waitfor also needs to modify the waitfor list, which means it needs the latch).

General Advice

This problem is a great example of something that more hardware can't fix - no amount of additional cores will help alleviate an instance-wide latch bottleneck. One of the most interesting parts of scaling up a SQL Server is all the wonderful new bottlenecks you discover - higher core counts (and workloads to drive those cores) make it easier to find concurrency bottlenecks (at lower core counts worker thread starvation will probably get you before you hit this latch).

If you are currently using SSB and think you may end up with hundreds of threads performing operations against SSB queues:

For low latency sensitivity (don't care about response time) - use receive + sleep
At low volume and high latency sensitivity - use waitfor-receive
If you need high volume and you have high latency sensitivity...look elsewhere

In many cases the scale and performance you need to achieve might require you to architect away from SSB:

Perfectly tuned, SSB cannot meet your requirements
You don't have (or want to build) the expertise needed to tune SSB
Other solutions give comparable or superior performance, the cost to migrate is low

These are especially true when you don't need some of the unique features SSB provides (multicast messaging, routing between multiple databases, poison message handling...). Those features come with a performance price!

The story of how we found ourselves with so much SSB I think is fairly typical. The pattern we adopted was a victim of it's own success, and so went from being used in a few places to being used everywhere. Only recently have these bottlenecks started to show up, and a gradual shift to other solutions is letting us break through successive scale barriers.

Some of those solutions involve switching to dedicated message queue product (e.g. RabbitMQ), and some have involved moving away from queues entirely (e.g. GRPC). Some of the more successful (measured as low implementation cost/high performance gain) have been switching queues out for tables (either regular or in-memory).

Saving and reporting on Pester Infrastructure test results in Log Analytics

Timothy Addison — Sun, 23 Sep 2018 00:00:00 +0000

Pester is a fantastic tool to test and monitor the status of your infrastructure. There are several libraries which leverage this directly (e.g. SQLChecks, DBAChecks, Operation Validation Framework), and a growing set of resources discussing how to leverage Pester in this fashion (including a Pluralsight course and a chapter in the Pester book). If you don't already have some tests you can run interactively (is my environment correct right now?) I'd suggest you start there first.

Once you've got your infrastructure tests running interactively you'll probably want to start automating them (is my environment correct every day?). And then at some point you'll probably find yourself asking questions like:

How long has this test been failing for?
How long does this test normally take to run?
When was the last time this test failed?
How is the health of my estate trending over time?

Once you've invested time into building out a library of Pester tests for your infrastructure, what you really want to do is analyse the Pester test results. There are various examples out there that discuss how to persist results to files, XML, SQL databases - but none of these options have the advantages that shipping to Log Analytics provides - which is what we'll discuss today. A few reasons why I think think sending your results to Log Analytics is the superior choice:

Extremely powerful query language (KQL)
Can be queried & consumed by a web browser, Power BI, REST API, Flow, Azure Monitor (for alerting)
Lives with the rest of your infrastructure logs (if you use the machine agents/Azure connectors/etc.)
Simplified security story (everywhere you run a tests posts to an HTTP endpoint - no fileshares/SQL databases/etc.)
No impact on the systems under test (how many DBAs have spent time performance tuning their monitoring databases :)?)
Great operating cost (free tier lets you log a lot of Pester results every month)

If you don't already have a Log Analytics workspace I did a fairly detailed write-up last month: Getting started with Log Analytics and PowerShell logging.

Pester Result Schema

While we could log an absolutely minimal object to Log Analytics, I've found that adding a little more structure is helpful for both debugging and analysing test results.

We'll be building the result objects in PowerShell, and then sending the objects to be stored in a table in Log Analytics. The below table shows the property names, as well as their column name in the table in Log Analytics. Columns are suffixed with their data type when created in Log Analytics, shown below as their LA Name (Log Analytics Name).

Property	LA Name	Description
Identifier	Identifier_s	Used to identify groups of Pester results, e.g. SQL Infra, AD Test, Fileshare Tests
BatchId	BatchId_g	Unique per batch of tests
InvocationId	InvocationId_g	Unique per call to Invoke-Pester
InvocationStartTime	TimeGenerated	The time the invocation started
InvocationEndTime	InvocationEndTime_t	The time the invocation ended
HostComputer	HostComputer_s	The computer running the tests
Target	Target_s	The target of the test (e.g. Host, SQL Instance)
TimeTaken	TimeTaken_d	The time taken for the test to execute
Passed	Passed_b	If the test passed or failed
Describe	Describe_s	The describe text for the test
Context	Context_s	The context text for the test, if present
Name	Name_s	The name text for the test
FailureMessage	FailureMessage_s	The error message for the test, if present
Result	Result_s	The result of the test

We're going to map the value of InvocationStartTime to the built-in field TimeGenerated. If no field is supplied, TimeGenerated defaults to ingestion time.

The great thing about the Data Collector API is that these fields are all optional, and so if you don't want to use the full schema you don't have to (perhaps your tests won't use context, or you won't care about host/target).

Some columns that deserve a little more explanation are BatchId and InvocationId.

Batches and Invocations

Most infrastructure tests I run tend to come in a format that looks something like the following (pseudocode):

for($thingToTest in $listOfThingsToTest) {
  Invoke-Pester -Script @{ TestParameters = $thingToTest }
}

The execution of the whole script would be a Batch. Every call to Invoke-Pester is a separate Invocation, which can have zero or more test results (technically an invocation has 0..N Describes, each of which has 0..N Contexts, each of which has 0..N Tests).

Being able to look at batches & invocations will let you detect issues like:

Incomplete batches (a hard-error knocked it out half-way)
Overall runtime vs. Invocation runtime (vs. test runtime)

Rather than collecting your results and posting them in one go, I would encourage you to post them after every Invoke-Pester call. There are times when your automation will fail, and having incomplete results will assist in telling you how far your batch got before failing (vs. having no results if you wait until the end to try and post them).

With batches and invocations our pseudocode now looks something like this (I've also included an example call to post data to Log Analytics):

$batchId = [System.Guid]::NewGuid()

foreach($thingToTest in $listOfThingsToTest) {
  $invocationId = [System.Guid]::NewGuid()
  $results = Invoke-Pester -Script @{ TestParameters = $thingToTest } -PassThru

  Export-LogAnalytics -Batch $batchId -Invocation $invocationId -Results $results
}

Note the usage of the PassThru switch - this is required to capture the results of the Pester tests in the $results variable.

The Export-LogAnalytics function is part of SQLChecks, though you can build something very similar with the example PowerShell on the Data Collector API page. You can see the definition of the function on GitHub - ExportLogAnalytics function. The source to the referenced function is also available on GitHub - Get-LogAnalyticsSignature.

Working with the Pester result object

The object returned from Invoke-Pester needs a bit of work to transform it into the schema we outlined above. The $results object contains a property TestResult, which is an array of result objects (one object for every test executed).

Each result object contains information about the Describe, Context, and Test, as well as the result (pass/fail) and timing information. We use the TestResult to build our array of PesterResult objects to sent to Log Analytics:

$results = Invoke-Pester -PassThru
$pesterResults = @()

foreach($testResult in $results.TestResult) {
  $pesterResults += [PSCustomObject]@{
  TimeTaken  = $testResult.Time.TotalMilliseconds
  Passed = $testResult.Passed
  Describe = $testResult.Describe
  Context = $testResult.Context
  Name = $testResult.Name
  FailureMessage = $testResult.FailureMessage
  Result = $testResult.Result
  }
}
# $exportLogAnalyticsArguments contains customerId, sharedKey, etc.
Export-LogAnalytics $pesterResults @exportLogAnalyticsArguments

The above example extracts the data from the Pester test results and no more (missing are things like BatchId, Target, etc.). Note that the above code is perfectly valid and can be used to quickly get started logging results.

A more complex example

A more complete example is shown below - this is taken from SQLChecks which iterates over configuration files and performs one call to Invoke-Pester (wrapped by Invoke-SQLChecks) per file being tested - in this case each file represents an instance of SQL Server.

$batchId = [System.Guid]::NewGuid()

$logAnalyticsArguments = @{
    CustomerId = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    SharedKey = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    LogType = "PesterResult"
    TimeStampField = "InvocationStartTime"
}

$configs = Get-ChildItem -Filter localhost.config.json -Recurse
foreach($configFile in $configs) {
    $config = Read-SqlChecksConfig $configFile.FullName
    $targetComputer = $config.ServerInstance

    $invocationStartTime = [DateTime]::UtcNow
    $results = Invoke-SqlChecks -Config $config -PassThru -Show None
    $invocationEndTime = [DateTime]::UtcNow

    # If the config has no tests, nothing to post in this batch
    if($results.Count -gt 0) {
        $pesterResults = @()
        foreach($testResult in $results.TestResult) {
            $pesterResults += [PSCustomObject]@{
                BatchId = $BatchId
                InvocationId = [System.Guid]::NewGuid()
                InvocationStartTime = $InvocationStartTime
                InvocationEndTime = $InvocationEndTime
                HostComputer = $env:computername
                Target = $TargetComputer
                TimeTaken = $testResult.Time.TotalMilliseconds
                Passed = $testResult.Passed
                Describe = $testResult.Describe
                Context = $testResult.Context
                Name = $testResult.Name
                FailureMessage = $testResult.FailureMessage
                Result = $testResult.Result
                Identifier = "SQLChecks"
            }
        }

        Export-LogAnalytics @logAnalyticsArguments $pesterResults
    }
}

A more complex example, simplified

Because this is a lot of code to write everywhere you deploy SQLChecks, it has been wrapped into a function - Invoke-SqlChecksToLogAnalytics, which means you can reduce the above example to the following:

$batchId = [System.Guid]::NewGuid()

$logAnalyticsArguments = @{
    CustomerId = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    SharedKey = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}

$configs = Get-ChildItem -Filter localhost.config.json -Recurse
foreach($configFile in $configs) {
    $config = Read-SqlChecksConfig $configFile.FullName
    Invoke-SqlChecksToLogAnalytics -Config $config -BatchId $batchId @logAnalyticsArguments
}

The definition of the Invoke-SqlChecksToLogAnalytics function shows how you might want to wrap your own infrastructure test library.

Querying results

Once you have some results in Log Analytics you can start to query them. You can get to the query interface via the Azure Portal, and once there to look at our query results we'd write:

PesterResult_CL
| order by TimeGenerated desc
| take 100

Showing recent batches

As there might be many kinds of Pester tests being shipped, we'll typically want to focus on a specific set - we'll use SQLChecks as an example again. The following code will find the most recent batch from the last 7 days, and show all results:

PesterResult_CL
| where TimeGenerated > ago(7d)
| where Identifier_s == "SQLChecks"
| top-nested 1 of BatchId_g by max(TimeGenerated)
| join kind= inner (
   PesterResult_CL
) on BatchId_g

While the most recent batch is a pretty common requirement, you may have different batch sizes (people running ad-hoc tests in the day are one example of smaller batches). One method I've used to find the most recent complete batch -is to look for queries that contain more than N results - I know my typical SQL checks have 900 tests, so the below query lets me filter out any small ad-hoc or incomplete batches:

PesterResult_CL
| where TimeGenerated > ago(7d)
| where Identifier_s == "SQLChecks"
| summarize testCount = count(), maxTime = max(TimeGenerated) by BatchId_g
| where testCount > 900
| top-nested 1 of BatchId_g by max(maxTime)
| join kind= inner (
   PesterResult_CL
) on BatchId_g

Showing failures only

When there are failures, you can quickly view details by filtering on the boolean column Passed_b. The below projects only the essential columns:

PesterResult_CL
| where TimeGenerated > ago(7d)
| where Identifier_s == "SQLChecks"
| summarize testCount = count(), maxTime = max(TimeGenerated) by BatchId_g
| where testCount > 900
| top-nested 1 of BatchId_g by max(maxTime)
| join kind= inner (
   PesterResult_CL
) on BatchId_g
| where not(Passed_b)
| project TimeGenerated, Describe_s, Name_s, FailureMessage_s

In this specific example the result means that the trace flags configured on the server differ from the expected trace flags by a count of one.

Showing batch aggregates

To look at the overall stats (tests, passed, failed) we can group by any set of columns - in the below example we're grouping by Target and Describe, and then ordering by the number of failed tests. This lets us quickly see which tests have failed and against what target.

PesterResult_CL
| where TimeGenerated > ago(7d)
| where Identifier_s == "SQLChecks"
| summarize testCount = count(), maxTime = max(TimeGenerated) by BatchId_g
| where testCount > 900
| top-nested 1 of BatchId_g by max(maxTime)
| join kind= inner (
   PesterResult_CL
) on BatchId_g
| summarize TestCount = count(), Passed = sumif(1,Passed_b), Failed = sumif(1,not(Passed_b))  by Target_s, Describe_s
| order by Failed desc

In this case it looks like we have some data file space issues in addition to the trace flag problem.

Showing test or machine history

We might want to look at how a single test is performing over the estate. The below query shows the status of the Data file space used Describe by percent success (0% = all tests failed, 100% = all tests passed), split by target. Note we multiply the count by 1.0 to turn it into a float, rather than an integer (which would floor our result to always 0 or 1).

PesterResult_CL
| where TimeGenerated > ago(30d)
| where Identifier_s == "SQLChecks"
| where Describe_s == "Data file space used"
| summarize Passed = sumif(1,Passed_b), Total = count() * 1.0 by Target_s, bin(TimeGenerated, 1d)
| extend PercentSuccess = Passed / Total
| project TimeGenerated, Target_s, PercentSuccess
| render timechart

The below graph shows an example of a few targets which have never had a failure (yay!), one target which was partially failing (note the Y axis starts at 0.5) for a long time and recently was fixed, and another which partially failed and then recovered.

Finding the longest-running Describe block

If you're looking to performance tune your infrastructure tests, you'll want to know where the time is being spent. This final example shows how you can find which one of the describe blocks is taking the longest time to run. The example uses the most recent batch and plots the time taken in milliseconds for each describe block.

PesterResult_CL
| where TimeGenerated > ago(7d)
| where Identifier_s == "SQLChecks"
| top-nested 1 of BatchId_g by max(TimeGenerated)
| join kind= inner (
   PesterResult_CL
) on BatchId_g
| top-nested 5 of Describe_s by sum(TimeTaken_d)
| order by aggregated_Describe_s desc
| render barchart

In this example checking for Duplicate Indexes dominates at almost 140 seconds.

Summary

Well done if you made it this far - if you're starting from scratch with your infrastructure testing there are a lot of steps needed to get here. The good news is that once you've gone through all this setup for your first set of tests, onboarding and analysing the results from subsequent tests is very easy.

By having all your Pester results stored in Log Analytics you're able to inspect the health of your estate now and historically very quickly, and can additionally share access to those test results directly (giving people the ability to write their own queries over your results), or create and share dashboards (perhaps leveraging the ability of Power BI to query Log Analytics). Some other options you have to leverage your results in Log Analytics include creating alerts with Azure Monitor (alert on Pester failures), or scheduling periodic reports with Flow (a daily summary of Pester results).

In the near future I'll be showing how you can use Pester to perform data validation checks too - the results will, of course, be shipped to Log Analytics for easy querying/monitoring/alerting.

Getting started with Log Analytics and PowerShell logging

Timothy Addison — Sun, 26 Aug 2018 00:00:00 +0000

Log Analytics is a fantastic place to ship, store, and analyse your logs. Whether they're coming from a linked Azure resource, machine agents, or you're posting them from your own applications and services, Log Analytics is a key part of Azure Management & Monitoring. Whether you're an IT Pro, working in devops, or an application developer - this platform and its capabilities are worth exploring and understanding.

Log Analytics was previously offered as part of the Operations Management Suite (OMS) bundling, though that labelling is in the process of being retired.

To get started you can create a free workspace which lets you ingest up to 5GB of data per month. Once the data is loaded there is no cost to query it, and it'll be retained for 31 days (you can up retention and ingestion limits as part of paid plan later).

This post will walk through creating a Log Analytics workspace, uploading some logs with PowerShell, and then querying them via the portal. To follow along you'll need an Azure subscription and the AzureRM PowerShell module - for installation instructions see the prerequisites section at the end of this post.

Creating the workspace

We'll create the workspace using PowerShell. If you prefer you can create the workspace via the Azure Portal.

To start you'll need to import the AzureRM PowerShell module and connect to your Azure account. If you have more than one subscription you'll need to select the one you want to work with.

Import-Module AzureRm
Connect-AzureRmAccount

# If you have more than one subscription select the one you want to create the workspace in
Get-AzureRmSubscription -SubscriptionName "Sub Name" | Select-AzureRmSubscription

Once connected to a subscription you can create a workspace. The below code creates it in the East US location, though there are a few other regions available. The script below also creates a resource group to store the workspace (though you could also specify an existing resource group).

The resource group name must be unique within your subscription, and the workspace name must be globally (not just your subscription/organisation) unique.

$resourceGroupName = "RG-LogAnalytics-Test"
$location = "eastus"
$workspaceName = "LogAnalyticsTestWorkspace"

New-AzureRmResourceGroup -Name $resourceGroupName -Location $location

$newWorkspaceParams = @{
    ResourceGroupName = $resourceGroupName
    Location = $location
    Sku = "standalone"
    Name = $workspaceName
}

New-AzureRmOperationalInsightsWorkspace @newWorkspaceParams

Note the SKU is actually PerGB2018 - the older SKUs (free, standard, premium, standalone, pernode) aren't supported in subscriptions created after April 2018. You can read more details in the announcement blog post. We specify the standalone SKU though it'll actually create with PerGB2018 (the AzureRM cmdlets don't yet support the PerGB2018 SKU).

The last thing we'll do is apply a data volume cap to ensure our workspace doesn't exceed the free tier's limit of 5GB/month. Currently this can't be done with a PowerShell cmdlet, so you'll need to use the portal.

The screenshot below shows what this looks like, the steps you need to take from the Log Analytics page are:

Select the Usage & Billing blade
Select the Data volume management tab
Turn the Daily Volume Cap ON
Set a volume which keeps you within your 5GB/month limit (e.g. 0.15GB/day)
Press OK to apply the settings

I've opened an issue to request adding support to set daily volume caps in the AzureRM PowerShell module.

Posting data to the workspace

To post data to the workspace you'll need one of the two workspace keys as well as the CustomerId (sometimes called the Workspace Id). You can get this information from the portal, or via PowerShell:

Get-AzureRmOperationalInsightsWorkspaceSharedKeys -ResourceGroupName $resourceGroupName -Name $workspaceName
(Get-AzureRmOperationalInsightsWorkspace -ResourceGroupName $resourceGroupName -Name $workspaceName).CustomerId

To send data to our workspace we're going to use the Log Analytics Data Collector API. This API lets us send a JSON payload which gets logged to a table in of our workspace. We're going to use the Export-LogAnalytics function (available in this GitHub Gist) to send the data.

Our test dataset will be a snapshot of currently running processes, including their name and handle count.

The Export-LogAnalytics function uses ConvertTo-Json to build the JSON payload, which by default attempts to serialize the object with depth 2. By building our own pscustomobject we have fine-grained control over exactly what schema our final table will have (and in most cases you won't want to log every property on the typically data-rich PowerShell objects you are working with).

$processes = Get-Process | ForEach-Object {
    [pscustomobject]@{
        ProcessId = $_.Id
        ProcessName = $_.ProcessName
        Handles = $_.HandleCount
        CollectionTime = [System.DateTime]::UtcNow
    }
}

$logAnalyticsParams = @{
    CustomerId = "From-Earlier-Step..."
    SharedKey = "From-Earlier-Step..."
    TimeStampField = "CollectionTime"
    LogType = "Process"
}

Export-LogAnalytics @logAnalyticsParams $processes

A few things to note:

The table name is based on the LogType argument, and always has _CL appended (for Custom Log)
Every property will get its own column in the final table, typed as either a string, boolean, datetime, guid, or double
The column name is suffixed based on the data type - _d for double, _t for datetime, etc.
If TimeStampField isn't specified the time of every record is based on ingestion time, otherwise the TimeGenerated property is set based on the field specified
- In our example TimeGenerated field will contain the value from the CollectionTime property, and CollectionTime will not appear as a field in the table
If an array of objects are passed to the function, each object is ingested as a separate row

For more information about the Data Collector API check the documentation

Viewing the data

Important note: There can be significant delays between you posting the data and it being available to query. This is known as ingestion delay, and the current SLA is 6 hours. Data typically appears within 10 minutes, and although your custom table (e.g. Process_CL) will appear immediately once you post data, the fields won't appear until the data is ingested. If no rows appear, wait a little while!

Once the data has been ingested you can navigate to the Log Analytics query interface via the Azure Portal (the Logs blade), or you can run the below PowerShell to get a direct link to query your workspace:

$subscriptionId = (Get-AzureRmContext).Subscription.Id
$uri = "https://portal.loganalytics.io/subscriptions/{0}/resourcegroups/{1}/workspaces/{2}" -f $subscriptionId, $resourceGroupName, $workspaceName
$uri

Once in the query portal you can write a query (if you used the link from PowerShell you'll need to open a new tab). The below example lists processes ordered by the number of handles:

ProcessLog_CL
| order by Handles_d desc
| project TimeGenerated, Handles_d, ProcessName_s

You can also deep-link queries using the URI we built earlier, appending the query you want to run (you'll need to URL encode the query if it contains spaces/line breaks/etc.):

"{0}?query=Process_CL" -f $uri

Clean-up

If you don't want to keep your workspace around you can tidy up by either removing the whole resource group (which deletes all resources in it), or just the workspace:

# Remove the whole resource group
Remove-AzureRmResourceGroup -ResourceGroupName $resourceGroupName

# Just remove the log analytics workspace
Remove-AzureRmOperationalInsightsWorkspace -ResourceGroupName $resourceGroupName -Name $workspaceName

You can also use the Azure portal to delete either of these.

Next steps

The world is your data-driven oyster now you've got a workspace configured! All you need now is some data - a great way to get started is by connecting your machine to log analytics and shipping some performance counters.

Once you've got some data you'll want to get to grips with the query language of Log Analytics - KQL (Kusto Query Language). This language is making an appearance in more and more products (Application Insights and Windows Defender Advanced Threat Protection are two particularly prominent ones), and is extremely powerful. The documentation home page for Log Analytics is a good place to start, or you can dive straight into the query language reference.

This really is only a small taste of what you can do when you start leveraging Log Analytics as the glue for your monitoring and alerting (which is whole other facet of the product to explore). The below image (taken from Microsoft's Log Analytics overview page) should give you plenty of ideas for how it can be used.

I hope you have as much fun querying your logs as I do!

Prerequisites

You can create a free Azure Subscription if you don't already have one. You'll need a Microsoft Account to do that. The subscription comes with a bunch of services that are free for 12 months, along with some that are free forever (including 5GB per month of Log Analytics).

If you want to use PowerShell to create your workspace and follow along with most of the examples in this post you'll need to install AzureRM PowerShell - the easiest way to do that is to execute the following from an elevated PowerShell prompt:

Install-Module -Name AzureRM

Full workspace creation script

The below script creates the resource group, workspace, and then prints the primary & secondary keys, as well as the URI for querying logs. You still need to visit the portal to set a daily cap if you want to avoid any charges.

Import-Module AzureRm
Connect-AzureRmAccount

$subscriptionName = "Tim Addison - MSDN"
Get-AzureRmSubscription -SubscriptionName $subscriptionName | Select-AzureRmSubscription

$resourceGroupName = "RG-LogAnalytics-Test" # subscription unique
$location = "eastus"
$workspaceName = "LogAnalyticsTestWorkspace" # globally unique

New-AzureRmResourceGroup -Name $resourceGroupName -Location $location

$newWorkspaceParams = @{
    ResourceGroupName = $resourceGroupName
    Location = $location
    Sku = "standard"
    Name = $workspaceName
}

New-AzureRmOperationalInsightsWorkspace @newWorkspaceParams

Get-AzureRmOperationalInsightsWorkspaceSharedKeys -ResourceGroupName $resourceGroupName -Name $workspaceName
(Get-AzureRmOperationalInsightsWorkspace -ResourceGroupName $resourceGroupName -Name $workspaceName).CustomerId

$subscriptionId = (Get-AzureRmContext).Subscription.Id
$uri = "https://portal.loganalytics.io/subscriptions/{0}/resourcegroups/{1}/workspaces/{2}" -f $subscriptionId, $resourceGroupName, $workspaceName
$uri

Export-LogAnalytics function

The below script contains the Export-LogAnalytics function and supporting Get-LogAnalyticsSignature function. They are both derived from the examples on the Log Analytics Data Collector API page.

# Adapted from https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-data-collector-api
Function Get-LogAnalyticsSignature {
    [cmdletbinding()]
    Param (
        $customerId,
        $sharedKey,
        $date,
        $contentLength,
        $method,
        $contentType,
        $resource
    )
    $xHeaders = "x-ms-date:" + $date
    $stringToHash = $method + "`n" + $contentLength + "`n" + $contentType + "`n" + $xHeaders + "`n" + $resource

    $bytesToHash = [Text.Encoding]::UTF8.GetBytes($stringToHash)
    $keyBytes = [Convert]::FromBase64String($sharedKey)

    $sha256 = New-Object System.Security.Cryptography.HMACSHA256
    $sha256.Key = $keyBytes
    $calculatedHash = $sha256.ComputeHash($bytesToHash)
    $encodedHash = [Convert]::ToBase64String($calculatedHash)
    $authorization = 'SharedKey {0}:{1}' -f $customerId,$encodedHash
    return $authorization
}

Function Export-LogAnalytics {
    [cmdletbinding()]
    Param(
        $customerId,
        $sharedKey,
        $object,
        $logType,
        $TimeStampField
    )
    $bodyAsJson = ConvertTo-Json $object
    $body = [System.Text.Encoding]::UTF8.GetBytes($bodyAsJson)

    $method = "POST"
    $contentType = "application/json"
    $resource = "/api/logs"
    $rfc1123date = [DateTime]::UtcNow.ToString("r")
    $contentLength = $body.Length

    $signatureArguments = @{
        CustomerId = $customerId
        SharedKey = $sharedKey
        Date = $rfc1123date
        ContentLength = $contentLength
        Method = $method
        ContentType = $contentType
        Resource = $resource
    }

    $signature = Get-LogAnalyticsSignature @signatureArguments

    $uri = "https://" + $customerId + ".ods.opinsights.azure.com" + $resource + "?api-version=2016-04-01"

    $headers = @{
        "Authorization" = $signature;
        "Log-Type" = $logType;
        "x-ms-date" = $rfc1123date;
        "time-generated-field" = $TimeStampField;
    }

    $response = Invoke-WebRequest -Uri $uri -Method $method -ContentType $contentType -Headers $headers -Body $body -UseBasicParsing
    return $response.StatusCode
}

Troubleshooting a slow application with Application Insights

Timothy Addison — Sun, 05 Aug 2018 00:00:00 +0000

Application Insights (AppInsights) is a fantastic instrumentation framework that with minimal/zero configuration will start giving you rich data about your application's performance.

We recently got some reports that one of our website solutions was 'slow' when developing locally, and as much as we'd like to turn to the DBA (which can often stand for Database Blamed Always...), with AppInsights we can be a little more rigorous.

From our starting point of 'it runs slow locally, I think it is the database' we'll figure out precisely how slow it is, and whether it really is the database or not.

Investigating with the portal

After navigating to the AppInsights resource that holds our development telemetry, we select the performance blade.

We prefer to split our telemetry so that the production environment has its own resource, and that all DevTest resources go to their own resource.

We can now to navigate to the time period where the application was misbehaving (slow), and then filter out all of the roles and instances we don't care about.

In our case we drilled down to select the frontend website application, and the development machine which reported the problem.

The performance blade gives an overview of the call count & duration of each endpoint (e.g. /Items/Detail, /Account/Login), and we can quickly see that a few endpoints had really poor performance - 4 minutes to load one page. Discrepancies between production and local development are to be expected (none of our development team run any 64-core machines), but 4 minutes is pretty extreme.

These are all server-side response times. AppInsights also supports client side monitoring.

From here we can either drill into that single slow call, or take a look at overall dependency performance.

Overall dependencies

AppInsights will track every outbound dependency made by the application, giving us access to individual calls (e.g. this call at time T took 12ms) but also aggregate summaries.

From the dependencies view we see a breakdown of every outbound call with the call count and average duration. You'll need to know what 'normal' looks like for your application, or at least have a rough idea of what to expect (volume of calls and expected response time). Looking at this view it was clear there was an issue with dependencies - call times I was expecting to be <10ms (in this environment) were running at almost 100ms.

Flipping between a machine that wasn't suffering the problems and the problematic machine revealed the 'slow' machine was almost 5-10x slower than the normal machine. You can be done via the portal (selecting a different instance), or by dropping into the Analytics portal and querying a few different machines:

let start=datetime("2018-07-30T15:00:00.000Z");
let end=datetime("2018-07-30T17:00:00.000Z");
dependencies
| where timestamp > start and timestamp < end
| where cloud_RoleInstance == 'SLOW_PC_HOSTNAME') // or 'FAST_PC_HOSTNAME'
| where (cloud_RoleName == 'Website'
| summarize count_=sum(itemCount), avg(duration), percentiles(duration,50,95) by target, name

Looking at averages as well as the median and 95th percentile shows us that this isn't some long-tail of slow requests pushing up the mean, but that all of our requests are slow. The problem is impacting all dependencies (e.g. GRPC), not just the database.

AppInsights doesn't capture GRPC dependencies out of the box, but it does capture SQL and HttpClient calls.

Looking at an individual request

Another way to troubleshoot is to examine what happened in the context of a single request. If a dependency is called as part of a request, AppInsights will link the two together. By selecting one of the slow endpoints in the Overall view we can then see a list of all the sample operations AppInsights has captured.

Clicking on any sample shows a timeline for the operation, including all the dependency calls associated with that operation. Picking a slow example (the 4 minute one) we can see again this isn't one slow dependency - they're all slow (and what you can't see below is a long vertical scrollbar!).

AppInsights by default turns every request into an operation. If you're instrumenting a windows service you'll need to create and track your own operations if you want to see them show up in the Overall view with dependencies tagged to their parent operation. See more in the Telemetry Correlation docs.

As with the overall view we can drop down to Analytics to query the raw data for a single operation:

dependencies
| where operation_Id == "oPerAtionId="
| summarize sum(itemCount), avg(duration) by target, name
| order by sum_itemCount desc

Note that we sum(itemCount) rather than count() because of automatic sampling. If there is no sampling this has no impact on your results, but if your data is being sampled and you don't sum(itemCount), your numbers will be off!

This analysis tells us the same story as the overall view - that all dependencies seem to be slow. This view additionally tells us that we might have an N+1 problem somewhere due to the number of dependency calls being made with each operation.

The verdict

AppInsights is fantastic for pinpointing issues, but it can't root cause for you (yet?). In this case I already had a pretty good idea of what the problem was, and we used AppInsights to confirm it.

The slow development machine was located on the other side of the Atlantic from the development services it was configured to talk to! While some of our engineering team are working on increasing the speed of light, the short term fix we implemented was to target services local to the development machine.

The examples above are just the tip of the iceberg when it comes to diagnosing application performance (whether through the portal, using workbooks, or via Analytics - you can quickly ask and get answers to some incredibly detailed questions).

AppInsights is moving pretty quickly (every few months a new feature or experience pops up), and in the last 12 months we've seen Log Analytics and AppInsights Analytics converge to support the Kusto Query Language (KQL), and so it is worth keeping an eye on the Application Insights Blog.

Ensuring your Describe Tags are unique in Pester tests

Timothy Addison — Sat, 21 Jul 2018 00:00:00 +0000

The name of each test in SQLChecks is used as both the setting name in the configuration files, and to tag the Describe block. After seeing the benefit of fine-grained control over test execution (from Claudio Silva's post dbachecks - a different approach...) this method of test invocation became the preferred way to leverage the SQLChecks library:

$config = Read-SqlChecksConfig -Path $sqlChecksConfigPath
foreach($check in (Get-SqlChecksFromConfig $config)) {
  # Note we invoke by -Tag $check - a test with no tag will never get invoked
  Invoke-SqlChecks -Config $config -Tag $check
}

There isn't yet a convention for how to name a test, and we've already had some tests built with similar sounding names - it is only a matter of time before we get a duplicate. To prevent duplicate tests accidentally getting checked in (and causing unusual/broken behaviour for consumers), I recently added a test that parses the test files and ensures that each tag is not only unique within the file, but globally within SQLChecks.

You can find the full test on GitHub, or read on for an explanation of how it is implemented. For a more thorough exploration of tests you can run on Describe blocks see SQLDBAWithABeard's blog Using the AST in Pester for dbachecks (which inspired this test), or the TechNet post learn how it pros can use the PowerShell AST.

Overview

At a high level the test:

Finds all of the Pester test files that are shipped with the module
Get the content of each file (we use -Raw to get the full text, not a collection of lines)
Parses each file so we can interact with them as objects (e.g. Function, Describe block, Operator) rather than strings of text
Finds every Describe block
Records the tag (if it exists) in an array of all tags
Loops through the array of tags and checks if any of them has a count greater than 1

Looping through the array and testing each tag gives the test a fairly useful output (Tag X is a duplicate) rather than something generic and unhelpful (There are duplicate tags). If your tests are going to fail you want them to be maximally helpful in debugging the failure.

Most of the requirements (get all the files, loop through the array) are fairly straightforward PowerShell - if we take out the parsing code the test looks like this:

Describe "Module test Describe tags are unique" {
  $tags = @()

  Get-ChildItem -Filter *.tests.ps1 -Path $PSScriptRoot\..\src\SQLChecks\Tests | `
    Get-Content -Raw | `
    ForEach-Object {
      #TODO: Add the tag to the array
  }

  foreach($tag in $tags) {
      It "$tag is a unique tag within the module" {
          ($tags | Where-Object {$_ -eq $tag}).Count | Should Be 1
      }
  }
}

Parsing PowerShell

Parsing PowerShell is a well-trodden path and we have some excellent tools available, with the Parser class allowing us to take a string and turn it into an Abstract Syntax Tree (AST). The tree structure gives us an incredibly rich object graph that we can interact with and query (so rather than writing a regex to find all Describe blocks, we can ask the AST to find all the Describe commands).

We build the AST by passing the content from our files to the ParseInput command (this is on the inside of the Get-Content -Raw | ForEachObject { block)

$ast = [Management.Automation.Language.Parser]::ParseInput(
    $_,
    [ref]$null,
    [ref]$null
)

The two [ref]$nulls are needed to satisfy the required parameters of ParseInput - in this case we don't care about capturing the tokens or any errors returned (see the ParseInput documentation for more details)

Once the AST is built we can then run a query to find all nodes that satisfy a set of predicates. In our case we want to find:

Every Describe command (Remember, Describe is a PowerShell function!)
Where there is a second parameter (we'll assume the first parameter is the description, e.g. Describe "Some Test")
Where that second parameter name is Tag

And once we have found every Describe command that satisfies those predicates, we want to take the fourth element (called the CommandElement) which will be the Tag parameter's value. Translated into PowerShell our query looks like this (remember the FindAll method can produce multiple results, so we have to extract the tag from each one):

$ast.FindAll({
          param($node)
          $node -is [System.Management.Automation.Language.CommandAst] -and
          $node.CommandElements[0].Value -eq "Describe" -and
          $node.CommandElements[2] -is [System.Management.Automation.Language.CommandParameterAst] -and
          $node.CommandElements[2].ParameterName -eq "Tag"
      }, $true) | ForEach-Object {
          $tags += $_.CommandElements[3].Value
      }

The FindAll functions takes a predicate function which should return $true if the node matches. The first line of our predicate (...-is [System...) limits our search to commands only (not comments, blocks, etc.).

You'll note we don't check there is a fourth command element - this would be invalid syntax (missing parameter value/no test block) and that sounds like another test we could write.

The complete test

Describe "Module test Describe tags are unique" {
  $tags = @()

  Get-ChildItem -Filter *.tests.ps1 -Path $PSScriptRoot\..\src\SQLChecks\Tests | Get-Content -Raw | ForEach-Object {
      $ast = [Management.Automation.Language.Parser]::ParseInput($_, [ref]$null, [ref]$null)
      $ast.FindAll({
          param($node)
          $node -is [System.Management.Automation.Language.CommandAst] -and
          $node.CommandElements[0].Value -eq "Describe" -and
          $node.CommandElements[2] -is [System.Management.Automation.Language.CommandParameterAst] -and
          $node.CommandElements[2].ParameterName -eq "Tag"
      }, $true) | ForEach-Object {
          $tags += $_.CommandElements[3].Value
      }
  }

  foreach($tag in $tags) {
      It "$tag is a unique tag within the module" {
          ($tags | Where-Object {$_ -eq $tag}).Count | Should Be 1
      }
  }
}

If you've not worked with an AST or parser before (or you have but not in PowerShell) some of this might look a little intimidating (it took me a while to grok it), but I'd encourage you to persevere as the pattern is incredibly powerful.

It's pretty cool to write Pester tests for your Pester tests.

Adding more tests

Once we have the pattern for parsing and inspecting a test we can add additional tests fairly easily - the below is a full example that will check if every test (Describe block) has a Tag parameter. The main difference to the previous test is we perform our checks per-describe (inside a context that specifies the file, so you can easily track down any failures). We've also moved the tests for parameter/parameter value out of the FindAll and onto the other side of a Should test (because we want to find all the describe blocks and check if they're well formed afterwards).

Describe "Every test has a tag" {
    Get-ChildItem -Filter *.tests.ps1 -Path $PSScriptRoot\..\src\SQLChecks\Tests | ForEach-Object {
        $content = Get-Content $_.FullName -Raw
        Context "$_" {
            $ast = [Management.Automation.Language.Parser]::ParseInput($content, [ref]$null, [ref]$null)
            $ast.FindAll({
                param($node)
                $node -is [System.Management.Automation.Language.CommandAst] -and
                $node.CommandElements[0].Value -eq "Describe"
            }, $true) | ForEach-Object {
                It "$($_.CommandElements[1]) has a tag" {
                    $_.CommandElements[2] -is [System.Management.Automation.Language.CommandParameterAst] -and
                        $_.CommandElements[2].ParameterName -eq "Tag" | Should Be $true
                }
            }
        }
    }
}

Which looks something like this:

Adding Pester tests to a PowerShell module and scheduling CI with AppVeyor

Timothy Addison — Mon, 25 Jun 2018 00:00:00 +0000

Adding Pester tests to a PowerShell module is probably one of the most valuable development activities you'll be able to perform, and I'd encourage you to do it early in your project. I left it until rather late with SQLChecks, and as a result have broken the module several times.

While some of breaks were definitely edge cases (and I don't think I have the the foresight to write a test that would have caught them), one of the most egregious errors caused SQLChecks to not export any functions at all. In this post I'll walk through the steps needed to do just that, which in brief are:

Create a Pester test for the module
Run the test locally, demonstrating that it will fail when the module doesn't export any functions
Schedule the test to run automatically every time a commit is pushed to the GitHub repo (using AppVeyor)
Display the build status and number of passing tests on the Github readme

And once we're done a quick glance at our readme will show:

Adding a Test

We're going to add the Pester tests to a new tests folder in the root of the repository. The module (including its manifest and all functions) are all in the src/SQLChecks folder.

The goal of our the first test is to ensure that the module exports something - not a big hurdle, but it's a start. The complete SQLChecks.Module.tests.ps1 file looks like this:

Import-Module $PSScriptRoot\..\src\SQLChecks -Force

Describe "Import-Module SQLChecks" {
  Context "Module Exports" {
    It "Should export at least one function" {
      @(Get-Command -Module SQLChecks).Count | Should BeGreaterThan 0
    }
  }
}

We can then verify SQLChecks exports a function by running Invoke-Pester in the tests folder:

Note that because SQLChecks contains several .tests.ps1 files as embedded resources, running Invoke-Pester from the root folder of the repository will generate a lot of false positives.

Breaking the module

We can also demonstrate what happens when we break our module import process by commenting out the line that dot-sources all of our functions:

$files = Get-ChildItem -Recurse -Filter *.ps1 -Path $PSScriptRoot/Functions

foreach($file in $files) {
    #. $file.FullName # This line shouldn't be commented!
}

Which leads to...

Although this is an extremely simple test we have created a repeatable and easy way to ensure (one form of) correctness. More importantly we've also created a process which anyone contributing to SQLChecks can use to verify they've not made things any worse.

Over time I plan to add more checks that will not only verify correctness, but also check formatting and conventions are being followed.

Now that we have a repeatable way of testing SQLChecks is correct, the next step is to automate that check...

Configuring Continuous Integration

AppVeyor is a continuous integration & continuous deployment service that allows us to react to some or all commits to our repository. We're going to use it to run Pester tests after each commit, though if you browse through the AppVeyor documentation you'll see it is capable of much more (build, test, and deploy software/artifacts).

AppVeyor is free for open source projects, and if you don't already have an account you can sign up for one with your GitHub credentials. From your account home you can create a new project and link it to your GitHub repository - AppVeyor will then start running its default build process every time you commit.

I'm using GitHub, but AppVeyor also supports projects hosted in many other providers - Gitlab, VSTS, and Bitbucket to name a few.

Out of the box AppVeyor doesn't support automatic discovery of Pester tests, so we'll use an appveyor.yml file (placed in the root of our repository) to tell it what pipeline to run after every commit:

Do nothing if all the files changed are markdown files (.md) - see commit filtering for more details
Ensure Pester is installed (using Chocolatey)
Install modules listed as required in the SQLChecks manifest (SqlServer and DBATools)
Don't run any build step (we have nothing to build)
Run Invoke-Pester on the folder containing our tests

This looks like this:

skip_commits:
  files:
    - "**/*.md"

install:
  - cinst pester
  - ps: Install-Module dbatools -AllowClobber -Force
  - ps: Install-Module SqlServer -AllowClobber -Force

build: off

test_script:
  - ps: Invoke-Pester -Path .\tests

If we trigger a build (either manually by clicking the 'New Build' button inside your AppVeyor project, or commit a change to the repository) we'll see something like this:

Recording test results in AppVeyor

At this stage the build is still marked as successful even if some tests fail. Build status is dependent solely on the output of each step of the pipeline (if any fail, the build fails). Even if 100% of tests fail, the Invoke-Pester command still completed successfully, and so the build passes. We need to let AppVeyor know one or more tests have failed by causing our test step to fail when tests fail. The Pester wiki has an example script we can use to do just that:

test_script:
  - ps: |
      $testResultsFile = ".\TestsResults.xml"
      $res = Invoke-Pester -OutputFormat NUnitXml -OutputFile $testResultsFile -PassThru
      (New-Object 'System.Net.WebClient').UploadFile("https://ci.appveyor.com/api/testresults/nunit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path $testResultsFile))
      if ($res.FailedCount -gt 0) { 
          throw "$($res.FailedCount) tests failed."
      }

By including a throw in our script, we cause the script step to fail, so the whole build will be marked as failed.

The other part of this script (uploading a file with WebClient) takes the results and uploads them to AppVeyor, which means along with each build's status (pass/fail), we also get access to the results of each test. As you can see below, the 'Always Fail' test did indeed fail:

Now that we've got the build status and tests working correctly, we can display them as badges using shields.io (and you can click the badge to view the most recent build/test results):

If you look at the URL you'll see how you can modify them to fit your own project (and if you check the main shields.io page you'll find they support a lot of other badges too).

[![Build Status](https://img.shields.io/appveyor/ci/taddison/SQLChecks.svg)](https://ci.appveyor.com/project/taddison/sqlchecks)
[![Test Status](https://img.shields.io/appveyor/tests/taddison/SQLChecks.svg)](https://ci.appveyor.com/project/taddison/sqlchecks/build/tests)

Next Steps

Once the CI process is in place it only gets more valuable as you increase the number of tests that can be executed for every commit. Some examples of how SQLChecks tests might be enhanced in the near future:

Check behaviour of commands without needing SQL Server by mocking certain cmdlets (e.g. Invoke-SqlCmd)
Ensuring code standards are maintained with PSScriptAnalyzer
Making sure I don't screw up the module manifest
Ensuring that the 'Config' family of commands all actually accept a Config parameter (see this GitHub issue)

And this is only scratching the surface - a lot of people have spent a lot of time testing pretty much anything with Pester and PowerShell (including things like SQL Server Infrastructure, perhaps with a module like SQLChecks...).

Setting sp_configure values with SQLChecks

Timothy Addison — Thu, 31 May 2018 00:00:00 +0000

As of v1.0 SQLChecks now contains the Set-SpConfig command that allows you to take a file that documents a server configuration (specifically sp_configure values) and apply that configuration to a server. The configuration file is the same one used by Pester tests (perhaps in combination with something like dbachecks), which means you now have a mechanism to document, test, and set your server's configuration.

In order to apply the configuration to a single server you would run the following PowerShell (note that SQLChecks configuration files contain the instance name, which is why we don't have to specify a server):

  Read-SqlChecksConfig "c:\configs\localhost.config.json" `
  | Set-SpConfig -Verbose

Running in verbose means that it'll output progress as it changes a value, as well as a summary as it finishes (x/y config values updated).

Note the command compares the configured value against the expected value - if the configured value is correct but the runtime value is wrong then this will neither fail the Pester tests, nor update the value when using Set-SpConfig.

You can easily apply changes to a whole estate of servers by using the below script, which will find every config file in a folder (or subfolder) and apply the sp_configure values to the servers.

Get-ChildItem -Filter *.config.json -Recurse `
  | Read-SqlChecksConfig `
  | Set-SpConfig -Verbose

Under the hood this command leverages the dbatools library, specifically the commands Get-DbaSpConfigure and Set-DbaSpConfigure. These commands both end up creating SMO objects which are pretty costly in terms of time compared to a hand-rolled T-SQL solution to check/set these values.

There is a GitHub issue around performance, and Microsoft have said to 'expect dramatic improvement in the coming months' in this UserVoice issue related to SMO. If the performance improvements don't arrive or are not dramatic enough, adding some custom commands will probably be worth doing (especially if you are checking a lot of values over a lot of servers).

Cleaning up database mail profiles and accounts

Timothy Addison — Tue, 24 Apr 2018 00:00:00 +0000

You know that dev server you've got lying around that has about half a dozen database mail profiles on? Maybe one account for every provider you've tested? Or maybe it's even a production server that has had its profile faithfully updated to a new account each time your SMTP server moves but never had the old ones cleared out?

After recently moving all of our servers to use SendGrid SMTP for sending out database mail I decided to perform some long overdue spring cleaning. In our case that included a couple of dev servers, as well as a production server from which you could divine the history of the company from the various database mail profiles and accounts it had.

Your servers might have a legitimate reason to contain multiple mail profiles/accounts, so only run the below script on a server if you are sure you want to remove every account/profile except the default.

declare @defaultProfileId int;
select @defaultProfileId = pp.profile_id
from dbo.sysmail_principalprofile as pp
where pp.principal_sid = 0x0 /* Guest */
and pp.is_default = 1;

if @defaultProfileId is null
begin
	;throw 50000, 'No default profile set', 1;
	return;
end

/* Delete non-default profiles and their account mappings */
delete from dbo.sysmail_profileaccount where profile_id <> @defaultProfileId;
delete from dbo.sysmail_profile where profile_id <> @defaultProfileId;

/* Remove orphaned accounts */
with cte as (
	select a.account_id
	from dbo.sysmail_account as a
	where not exists (
		select *
		from dbo.sysmail_profileaccount as pa
		where pa.account_id = a.account_id
	)
)
delete from cte;

For an example script which configures the default profile and account to use a SendGrid SMTP profile, see this gist. Plug in in your SendGrid API key) and you're good to go.

Better expenses with Monzo and PowerShell

Timothy Addison — Sun, 22 Apr 2018 00:00:00 +0000

While making submitting expenses fun is probably impossible, we can leverage Monzo and PowerShell to start automating away some of the manual steps involved. In my specific case I need to submit monthly expenses with receipt support and categorise each expense. Combine Monzo's tagging, image support, and CSV export with PowerShell's ability to...well do pretty much anything, and we have a workable solution!

Assuming I've correctly tagged all my transactions with the #expense tag I can then take the monthly export (available from the spending tab) and get my expense submission mostly prepared with a single PowerShell command:

. c:\src\blog-monzo-expenses\Expenses.ps1

$monzoExport = "c:\expenses\monzoexports\March2018.csv"
$exportFolder = "c:\expenses\submissions\March2018"

Export-ExpenseFromMonzo -MonzoExport $monzoExport -ExportFolder $exportFolder

This creates a CSV that contains transactions tagged with #expense in the Monzo account currency, downloads the first image associated with each transaction (and names it to match the line in the CSV), and maps the Monzo category to the internal expense reporting category.

A little copy-paste later and I'm done. Read on for details on how it works, and where you can customise the data you return. If you want to use this on your own data (or try it out on some sample data) you can get everything you need in this example GitHub repo.

How the script works

The script:

Gets all the entries in the Monzo CSV export
Filters out anything that doesn't contain the tag #expense
Downloads the first image for each transaction (if one exists), and renames it to match the expense line (1,2,3...)
Maps the Monzo category to an internal category
Writes the expense line to a CSV, including ID, Date, Currency, Amount, Category, and if a receipt is present

Let's go through some of those steps in more detail. First of all - importing the CSV and filtering. A tag in Monzo is stored in the notes field.

$entries = Import-Csv -Path $MonzoExport
$expenseEntries = $entries | Where-Object { $_.notes -like "*#expense*" }

We then iterate over every entry and first of all check to see if there is an image. The Monzo format exports the images in an array in the form [ImageUrl,Image2URL]. We only grab the first image if there is more than one, and save it to disk.

The images are uploaded to S3 buckets with no extension, so we use the content-type to map to the appropriate extension (e.g. image/jpeg -> .jpg). While testing I've used images hosted in GitHub, which don't return a content-type - as such I've added .jpg, which always gets a photo viewer to launch.

$hasReceipt = "No"
$imagePathWithoutExtension = $exportFolder + "\Receipt_$expenseId"
if($expense.receipt -match "\[([^,]+).*\]")
{
    $expenseImage = Invoke-WebRequest -Uri $Matches[1] -OutFile $imagePathWithoutExtension -PassThru
    $extension = Get-ExtensionFromContentType -ContentType $expenseImage.Headers.'Content-Type'
    if($extension -eq $null)
    {
        # It'll get a photo viewer to launch
        $extension = "jpg"
    }
    Rename-Item -Path $imagePathWithoutExtension -NewName "Receipt_$expenseId.$extension"
    $hasReceipt = "Yes"
}

We then add an expense object to an array we'll later export. Whatever properties this object has will end up in the CSV, so if you wanted to add extra data (e.g. the whole notes, the merchant name) this is where you'd do it.

Note that we multiply the amount by -1, as debits (charges) show up as negative amounts in the transaction feed, whereas you'll typically report expenses as a positive amount.

$outExpenses += [pscustomobject]@{
    Id = $expenseId
    Date = [DateTime]::Parse($expense.created).ToString("yyyy-MM-dd")
    Type = Get-ExpenseTypeFromCategory $expense.category
    Currency = $expense.currency
    Amount = ([double]$expense.amount) * -1
    HasReceipt = $hasReceipt
}

Finally we export the CSV into the folder we've been saving the images into.

$outExpenses | Export-Csv -Path "$exportFolder\Expenses.csv" -NoTypeInformation

Using this on your data

The only thing you need to get started with your own data is the Expenses.ps1 file from the sample repo. Modify the PowerShell script below to include:

The path to your Expenses.ps1 file
The path to your Monzo CSV
The folder you want to save the export in

. c:\path\to\Expenses.ps1

$monzoExport = "c:\src\blog-monzo-expenses\Example\March2018Export.csv"
$exportFolder = "c:\temp\expenses\March2018"

Export-ExpenseFromMonzo -MonzoExport $monzoExport -ExportFolder $exportFolder

Hopefully you can see how you can easily take the general idea and customise it to fit whatever process you follow. Some examples of things I've played around with:

Tags to delineate trips (e.g. #tripNYC)
Tags to extract extra metadata (e.g. #E-Client-12345)
Caps (e.g. don't expense more than $50 in food/day)
Time zone offsets ()

Appendix: Sample Monzo export

id,created,amount,currency,local_amount,,local_currency,category,emoji,description,address,notes,receipt
tx_00001ABCDEFGHIJKLMNOPQ,2018-03-30 17:32:57 +0000,-25,GBP,-25,1,GBP,general,,,,,
tx_00002ABCDEFGHIJKLMNOPQ,2018-03-29 19:33:45 +0000,-20,GBP,-20,1,GBP,general,,,,,
tx_00003ABCDEFGHIJKLMNOPQ,2018-03-29 19:33:06 +0000,-13.3,GBP,-13.3,1,GBP,general,,,,,
tx_00004ABCDEFGHIJKLMNOPQ,2018-03-24 15:21:47 +0000,-5.5,GBP,-5.5,1,GBP,general,,,,,
tx_00005ABCDEFGHIJKLMNOPQ,2018-03-17 11:59:09 +0000,-19.5,GBP,-19.5,1,GBP,groceries,,,,#expense,
tx_00006ABCDEFGHIJKLMNOPQ,2018-03-11 16:11:55 +0000,-67.99,GBP,-67.99,1,GBP,entertainment,,,,#expense,
tx_00007ABCDEFGHIJKLMNOPQ,2018-03-10 01:03:26 +0000,-5.28,GBP,-7.27,1.376893939,USD,shopping,,,,#expense,[https://github.com/taddison/blog-monzo-expenses/raw/master/Example/2327734421_7087f8f47e_z]

Improving database Pester tests - per-database checks

Timothy Addison — Sun, 08 Apr 2018 00:00:00 +0000

When we first started putting tests together for SQLChecks we naively/optimistically thought we'd mostly be seeing a sea of green, with failures being rare. This influenced the way we developed 'database' tests, so that when you test an instance for 'databases with files too full', the test gives you a pass/fail for the entire instance.

This is fine when the test passes, but as soon as it fails it is spectacularly unhelpful in figuring out what broke.

Arriving in the morning to discover one (or more) databases on an instance have a problem isn't particularly actionable, and so I've recently started to move all SQLChecks tests over to per-database, which is a lot more helpful.

The rest of this post covers what the changes looked like, and talk a little more about the benefits of structuring tests this way.

Test changes

The original test is shown below, which produces a single result for the whole instance.

Describe "Data file space used" -Tag MaxDataFileSize {
    foreach($config in $configs) {
        $serverInstance = $config.ServerInstance
        $MaxDataFileParams=@{
            ServerInstance = $serverInstance
            MaxDataFileSpaceUsedPercent = $maxDataConfig.SpaceUsedPercent
            WhiteListFiles = $maxDataConfig.WhitelistFiles
        }

        Context "Testing for data file space usage on $serverInstance" {
            It "$serverInstance has all databases under Max DataFile Space Used" {
                @(Get-DatabasesOverMaxDataFileSpaceUsed @MaxDataFileParams).Count | Should Be 0
            }
        }
    }
}

In order to move the test to report per-database:

The Get-DatabasesOverMaxDataFileSpaceUsed function is replaced with a new one that no longer iterates over all databases in the SQL query, and instead executes against a single database, returning all files that are larger than the space used configuration value
A list of database to run the check against is obtained by using Get-DatabasesToCheck, ignoring any replica databases
A new test (It) is executed for each database - note the database name is updated in the hashtable each iteration

The formatting of the Context/It block was also changed to support easier consumption in the dbachecks Power BI dashboard (which expects instance name to be in specific places).

Describe "Data file space used" -Tag MaxDataFileSize {
    foreach($config in $configs) {
        $serverInstance = $config.ServerInstance
        $spaceUsedPercentLimit = $maxDataConfig.SpaceUsedPercent
        $MaxDataFileParams=@{
            ServerInstance = $serverInstance
            MaxDataFileSpaceUsedPercent = $spaceUsedPercentLimit
        }

        $databases = Get-DatabasesToCheck -ServerInstance $serverInstance -PrimaryOnly

        Context "Testing for data file space usage on $serverInstance" {
            foreach($database in $databases) {
                It "$database files are all under $spaceUsedPercentLimit% full on $serverInstance" {
                    $MaxDataFileParams.Database = $database
                    @(Get-DatabaseFilesOverMaxDataFileSpaceUsed @MaxDataFileParams).Count | Should -Be 0
                }
            }
        }
    }
}

Note that these tests have had code removed that isn't relevant to this example - you can see the full test in the GitHub source.

Appendix: Test code

The script to run a single test is shown below, and has an ad-hoc config built to test no database file on localhost is more than 90 percent full.

Import-Module .\src\SQLChecks -Force

$configData = @'
{
    "ServerInstance": "localhost"
    ,"MaxDataFileSize": {
        "SpaceUsedPercent": 90
    }
}
'@
$config = $configData | ConvertFrom-Json

Invoke-SqlChecks -Config $config -Tag MaxDataFileSize

Pattern - SQL Server as a shared cache for expensive stored procedures

Timothy Addison — Wed, 21 Mar 2018 00:00:00 +0000

The following scaling rules will take you a long way if you are supporting an environment with a SQL Server datastore:

Cache as much as you can in the application tier
Offload as much computation into the application as possible
Minimise the work high-volume queries have to do
Limit the number of transactions you do, and the work each transaction does

Knowing when to ignore (or even break) these rules is what keeps the job interesting.

The rest of the post will walk through a generic pattern to cache stored procedure results that vary by parameter, including the logic needed to expire, clean up, and evaluate the cache. This solution has been battle tested in production with a fairly expensive procedure called concurrently from multiple application nodes (for dozens of different parameter combinations).

The environment that motivated this work already aggressively caches results in the application tier - the specific motivation to cache the results in SQL came from the number of application nodes increasing. Building a shared cache service [or introducing something like Redis/Memcached] is a non-trivial engineering project, and the SQL CPU pressure this proc caused was significant. Other options (e.g. incremental cache updates/aggregation in the application tier) were also judged to be significant projects (or at least, more significant than caching it in the database!).

If you want to see the full example, you can check the complete source on GitHub. Before deploying into production I strongly suggest reading through the entire post for caveats and trade-offs.

The problem

For this worked example we're going to use a procedure that gets the top selling products for a given category - something like Amazon's category page which shows the top sellers.

create procedure dbo.GetTopSellingProducts
	@categoryId int
as
begin
	select	top(20)
			p.ProductId
			,sum(s.TotalSalePrice) as TotalSalePrice
	from	dbo.Sales as s
	join	dbo.Product as p
	on 		p.ProductId = s.ProductId
	where	p.CategoryId = @categoryId
	and		s.SaleDateTime > getutcdate() - 1
	and		p.AvailableStock > 0
	group by p.ProductId
	order by sum(s.TotalSalePrice) desc;
end

While this procedure might only take 2 seconds to run (and be subsequently cached for a few minutes), we're potentially calling it _categories _ nodes* times per cache duration. If we assume our cache duration is 5 minutes, and we have 100 categories and 10 nodes, that comes to 400 CPU-seconds per-minute, 10x what it could be.

In a perfect world deploying this solution would always reduce the time spent by a factor of number of nodes. The time spent on managing the cache is negligible, and the places where you lose perfect scaling tend to come from concurrent calls - this was an explicit design decision where preferring the proc to execute and cache multiple copies of the result set was preferable to a solution that uses locking to ensure only one session could update the cache at a time.

Cache solution

The solution comprises of a few different pieces:

A master control table to toggle caching on/off
A control table per-proc to keep track of what is available in the cache
A results table per-proc which acts as the cache
A clean-up procedure, which can be used to periodically clean the results
The stored procedure itself, which is augmented with caching logic

The reasons why there exist control tables as well as results tables are discussed in more depth in the Tradeoffs and design decisions section below.

Tables

create schema DBCache authorization dbo;
go
create table DBCache.MasterControl
(
	CachedEntity varchar(255) not null
	,IsEnabled bit not null
	,constraint PK_MasterControl
		primary key clustered (CachedEntity)
);
go
create sequence DBCache.SEQ_Control as int
	start with 1
	increment by 1
	cycle;
go
create table DBCache.dbo_GetTopSellingProducts_Control
(
	CacheDateTime datetime2(3) not null
	,ExpiryDateTime datetime2(3) not null
	,Param_CategoryId int not null
	,UseCount int null
	,Id int not null
	,constraint PK_dbo_GetTopSellingProducts_Control
		primary key clustered (Id)
	,index IX_dbo_GetTopSellingProducts_Control_ExpiryParams
		nonclustered (Param_CategoryId, ExpiryDateTime)
);
go
create table DBCache.dbo_GetTopSellingProducts_Results
(
	ControlId int not null
	,ProductId int not null
	,TotalSalePrice money not null
	,index CIX_dbo_GetTopSellingProducts_Results
		clustered (ControlId)
);
go

Each stored procedure that needs to be cached has a control/result table that are unique to that stored procedure. Specifically, we capture all parameters in the control table, and all output columns in the results table. The control table must be indexed based on both the parameters of the procedure and the expiry date to support efficient lookups (answering the question 'is there a valid cached result for my parameter set?').

Ensure the types and nullability of these columns match what the stored procedure already uses/produces. A great way to get the type information of the result set is to use sp_describe_first_result_set

Stored Procedure

The stored procedure shown above changes drastically when we add caching. The complexity tradeoffs are discussed in more detail in the sections below.

The generalised stored procedure flow for a procedure that has caching looks something like this:

Lookup the caching status from the master control table (on/off)
If caching is enabled, check to see if there is a valid cache entry (same parameters, expiry date in the future)
- If that record exists, return the results from the cache table
If the cached record doesn't exist, run the procedure logic and capture the final results in a temporary table
If caching is enabled, attempt to populate the cache
- First of all insert the results from the temp table
- Insert the control record and set the expiry date
- Keep a record of the newly inserted cache Id
- If any errors are hit, null out the variable holding cache Id
If the cache Id isn't null, serve results from the cache
If the cache Id is null, serve results from the temp table

This procedure features an order by, and so the order by has to be repeated everywhere results are returned to the client.

create or alter procedure dbo.GetTopSellingProducts
	@categoryId int
as
begin
	set nocount on;

	declare @DB_CACHE_STATUS bit = 0
			,@CACHE_DURATION_MINUTES int = 5;

	select	@DB_CACHE_STATUS = mc.IsEnabled
	from	DBCache.MasterControl as mc
	where	mc.CachedEntity = 'dbo.GetTopSellingProducts';

	declare @cacheControlId int;

	if @DB_CACHE_STATUS = 1
	begin
		select top(1) @cacheControlId = c.Id
		from DBCache.dbo_GetTopSellingProducts_Control as c
		where	c.ExpiryDateTime >= getutcdate()
		and		c.Param_CategoryId = @categoryId;

		update DBCache.dbo_GetTopSellingProducts_Control
			set UseCount = UseCount + 1
		where Id = @cacheControlId;
	end

	if @cacheControlId is not null
	begin
		select r.ProductId
			  ,r.TotalSalePrice
		from DBCache.dbo_GetTopSellingProducts_Results as r
		where	r.ControlId = @cacheControlId
		order by r.TotalSalePrice desc;

		return;
	end

	select	top(20)
			p.ProductId
			,sum(s.TotalSalePrice) as TotalSalePrice
	into	#results
	from	dbo.Sales as s
	join	dbo.Product as p
	on 		p.ProductId = s.ProductId
	where	p.CategoryId = @categoryId
	and		s.SaleDateTime > getutcdate() - 1
	and		p.AvailableStock > 0
	group by p.ProductId
	order by sum(s.TotalSalePrice) desc;

	if @cacheControlId is null and @DB_CACHE_STATUS = 1
	begin
	begin try
		set @cacheControlId = next value for DBCache.SEQ_Control;

		insert into DBCache.dbo_GetTopSellingProducts_Results
		(
			ControlId
			,ProductId
			,TotalSalePrice
		)
		select	@cacheControlId
			   ,r.ProductId
			   ,r.TotalSalePrice
		from	#results as r;

		insert into DBCache.dbo_GetTopSellingProducts_Control
		(
			Id
			,CacheDateTime
			,ExpiryDateTime
			,Param_CategoryId
			,UseCount
		)
		values
		(
			@cacheControlId
			,getutcdate()
			,dateadd(minute,@CACHE_DURATION_MINUTES,getutcdate())
			,@categoryId
			,1
		)
	end try
	begin catch
		set @cacheControlId = null;
	end catch
	end

	if @cacheControlId is not null
	begin
		select r.ProductId
			  ,r.TotalSalePrice
		from DBCache.dbo_GetTopSellingProducts_Results as r
		where	r.ControlId = @cacheControlId
		order by r.TotalSalePrice desc;
	end
	else
	begin
		select r.ProductId
			  ,r.TotalSalePrice
		from #results as r
		order by r.TotalSalePrice desc;
	end
end

Clean-up job

The clean-up job implements the following logic:

Disable caching for the stored procedure in the master control table
Wait for 45 seconds
Set lock_timeout to 1 second
Try to truncate the table
If we weren't successful, try again up to 5 times
Mark all cache entries as expired if they are in the future
Re-enable caching
If we weren't able to truncate the table, throw an error

The time to wait depends on your environment - if your calling application has a timeout of 30 seconds, then in theory 31 seconds is long enough. In this case we've deployed our wait as timeout x 1.5

create or alter proc DBCache.dbo_GetTopSellingProducts_CleanUp
as
begin
	set nocount on;

	declare @CONTROL_NAME varchar(255) = 'dbo.GetTopSellingProducts'
			,@truncatedResultsTable bit = 0;

	update DBCache.MasterControl
		set IsEnabled = 0
	where	CachedEntity = @CONTROL_NAME;

	waitfor delay '00:00:45';

	set lock_timeout 1000;

	declare @retryCount int = 0
			,@MAX_RETRY_COUNT int = 5;

	while (@retryCount < @MAX_RETRY_COUNT and @truncatedResultsTable = 0)
	begin
		begin try
			truncate table DBCache.dbo_GetTopSellingProductsResults;
			set @truncatedResultsTable = 1;
		end try
		begin catch
			set @retryCount += 1;
		end catch
	end

	update DBCache.dbo_GetTopSellingProducts_Control
		set ExpiryDateTime = getutcdate()
	where	ExpiryDateTime > getutcdate();

	update DBCache.MasterControl
		set IsEnabled = 1
	where	CachedEntity = @CONTROL_NAME;

	if @truncatedResultsTable = 0
	begin
		;throw 50001, 'Failed to clean up results table for dbo.GetTopSellingProducts', 1;
	end
end
go

Trade-offs and design decisions

Adding caching in the database is fairly straightforward in theory - we store a copy of the results somewhere and re-use them if they are available (our early prototypes looked very similar to the approach outlined by this post from 2013). Some of the reason's that the caching solution deviated from the simple approach are documented below.

Some of these are specific to the environment the solution is being deployed in, and others are driven by the specific workload we were optimising for. Evaluate your own needs carefully before deploying some or all of this solution into production.

No lock

Not to be confused with nolock! There exist multiple places in this solution where we could have attempted to coordinate access to resources (like results or the control table). We tried a few experiments, and none of them showed sufficient promise to convince us the additional complexity/risk (of long-held locks) was worth it. Our environment is highly transactional and so we opted to avoid any explicit locking, and prefer to duplicate work rather than serialise to perform work only once.

Consider the example when the cache has expired and ten nodes call the procedure - all ten will do the work to populate the cache. This is a fairly extreme example, and in practice we never see this many duplicates (on occasion we do get some double-work).

Being able to easily detect double work was a key driver for instrumentation.

Instrumentation

The caching solution features a UseCount column, which allowed us to monitor the efficacy of the cache when deployed, and verify it was actually delivering value (alongside other measures from QueryStore and application monitoring). The impact of this logging is negligible, and provides us with additional data about how the applications are calling the procedure (as we also know what parameters are being used).

Although a requirement of our implementation, the fact we log every time the cache is populated also allows us to trivially monitor when 2 or more nodes populate the cache at the same time. This is how we're able to confidently say the issue of double-work isn't impacting us.

Adding to the same database vs. a separate database

Our setup features availability groups, and we currently have no IO or networking issues. The benefit of placing the solution in the same database meant we had a much simpler deployment/failover/recovery story.

The main concern deploying it into the primary database is the size of the cache - an early version was deployed to dev without a schedule for the clean-up job - the next morning we woke up to a hot mess of database space alerts.

This example is a very narrow result set with only 20 rows. In production we have deployed this solution on result sets in the 10,000+ rows range with 5+ columns. Being able to benchmark zero impact on our IO latencies was key to our decision to go ahead with a deployment in our primary AG database.

Cache control table

The simplest solution we tried was to store a single copy of the results for any given parameter. Our requirement for no locks as well as clear instrumentation quickly rendered this approach unsuitable.

Adding a second table allowed us to adopt a pattern whereby we'd populate the results before populating the control table. Early implementations populated the control table and then the results table, and under high load/error conditions it was possible for a procedure to think a cached result was ready (as it had checked the control table), only to encounter an empty results table.

Another advantage with a second table was the ability to keep the indexing on the results table very simple, and only have to worry about the parameter-based index on the control table.

Finally, supporting an empty result set isn't possible when only using a results table. Consider a product with no sales - the correct result set has no rows, and the only way to cache that is with a second table in addition to your results table.

Clean-up

Storing expired results meant that clean-up became a critical issue. Initial attempts to run deletes were fairly slow (not to mention expensive compared to a truncate), so we decided that we had to do the work required to support truncating the results table.

The requirement to truncate was the motivation for the master control table - without some kind of coordination we couldn't find a way to safely truncate while the procedures were running. Stopping the procedures being called wasn't possible, but stopping the caching system was something we could support (the cache is a feature which aids performance - the system can run without it, albeit with reduced headroom).

Setting the right frequency of the clean-up job depends heavily on how much data is generated and how much space you can dedicate to the cache solution.

TempDb usage for staging results

This is definitely a tradeoff you should only consider if you are very comfortable with the current state of your tempdb usage, and your ability to monitor it. I'd suggest both tracking global tempdb metrics as well as query-level wait statistics, to ensure you're not unduly bottlenecking on tempdb

The primary motivator for the results table is protection against errors - on encountering any kind of error inserting to the results table we're faced with the option of re-running the expensive logic (or even worse returning nothing!). By staging the data we always have something to return. This has saved us on at least one occasion when a data type change in a base table was not propagated to the cache results table.

A second benefit of this approach is we only need to have one copy of the logic in the procedure. As we support running with DBCache enabled and disabled, an approach which doesn't feature a temp table typically featured a copy of the logic to populate the cache or return the results directly.

Implementing caching

A quick summary of what is needed to cache a procedure

Create the schema, sequence, and master control table
Create the control and results table for your procedure
Create the results clean-up procedure for your procedure
- Schedule the clean-up job to run
Modify the procedure to include caching
- Until you insert a row in the master control table caching is disabled
Insert a row in the master control table and turn caching on

The changes to the base proc are fairly intricate, but after you've done it a few times you'll have the pattern down.

There are a few scenarios when you'll probably want to think twice about implementing this solution:

Huge number of parameters/very large parameters (varchar(max) anyone?)
Proc is called from a small number of nodes (implement a simpler caching solution)
IO or TempDB pressure
Space pressure

Future Ideas

Some additional ideas that have been considered/discussed. The further down this rabbit hole we go the more attractive that shared cache service starts to look!

Adaptive cache - vary duration by parameter, current load
Cache bypass - maybe some parameters are cheap enough to not need caching/only cache expensive parameters
Smart expiry - if data hasn't changed enough don't expire the cache - leverage e.g. Timestamp/CDC
Rotating partition scheme - truncating trailing partitions and caching into a fresh partition makes the clean-up job more complex but eliminates the need to pause (cache results now keys on PartitionId, ControlId)

Partitioned Clustered Columnstores - Mind your deltastores!

Timothy Addison — Sat, 10 Mar 2018 00:00:00 +0000

We use an hourly partitioning scheme coupled with a clustered columnstore to support ingestion and short-term retention of various telemetry data. A fairly common pattern is for us to deploy an hourly partition schema based on the InsertDate of data (a column with a default of getutcdate()).

One thing we've had to keep an eye on when using this kind of pattern is the number of open deltastores that can get left behind as inserts 'move on' to a new partition.

Hourly partitions are used as they are also the unit of truncation as data ages out in our environment. Due to the pain of the 'dangling deltastores' (as well as high partition counts impacting DMV usage) we're looking at making daily the smallest partition size we support.

When partitioning a clustered columnstore table you will have one or more deltastores per partition. The largest I've seen so far is 20 open in a single rowgroup (with row counts ranging from 1M to 10k), though 4 is about average for this particular environment.

These open deltastores have a number of drawbacks:

Increased size on disk for uncompressed rows
Lack of rowgroup elimination for the rows in the deltastores
Reduced query performance on queries which work in the rowstores

The rest of this post will cover these in a bit more detail, as well as how to see if you're impacted and what to do about.

Increased size on disk

The first example is best shown by a table which had escaped any kind of index maintenance:

The deltastores might be less than 20% of the rows, but they're more than 80% of the space used for the table! In the example above compressing the open rowgroups reduced the table from 500GB to 150GB.

Lack of rowgroup elimination

Rowgroup elimination is the ability of a query against the columnstore to skip entire rowgroups based on the metadata about min/max values stored for each column segment in the rowgroup (see sys.column_store_segments). When querying our telemetry table for a single record in the last day (already leveraging partition elimination to limit our query), we'll also supply a predicate for a ServerId we know doesn't exist. This is beyond the max value that should exist in any rowgroup.

select top 1 *
from dbo.Telemetry as t
where t.ServerId = 99999999
and t.InsertDateTime>= cast(getutcdate() - 1 as datetime2(3));

Partition elimination won't work if your data types are more precise than the column you're partitioning on. In our case using getutcdate() would have been converted to datetime2(7), more precise than our InsertDateTime column which is datetime2(3), resulting in a full table scan [ouch]. Segment elimination has no such qualms about data type precision!

Table 'Telemetry'. Scan count 101, logical reads 1548578, physical reads 0, read-ahead reads 1363208, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Telemetry'. Segment reads 0, segment skipped 107.

 SQL Server Execution Times:
   CPU time = 7672 ms,  elapsed time = 9444 ms.

The statistics of this query show that every segment was skipped (they were successfully eliminated), yet we spent 10 seconds doing an awful lot of reads. These reads were against the deltastores, as SQL Server had to check every single row to ensure none of them were for ServerId 99999999.

Reduced query performance

In addition to a lack of rowgroup elimination, there are additional performance issues when operating against deltastores. Although deltastore scans do seem to benefit from batch mode, they don't benefit from aggregate pushdown.

Take the following query:

declare @cutoff1 datetime2(3) = '20180309';
declare @cutoff2 datetime2(3) = '20180310';

select count(*)
from dbo.Telemetry as t
where t.InsertDateTime >= @cutoff1
and t.InsertDateTime < @cutoff2;

The results when executed against our table with many open deltastores:

Table 'Telemetry'. Scan count 114, logical reads 1589897, physical reads 235, read-ahead reads 1588889, lob logical reads 79781, lob physical reads 24, lob read-ahead reads 246172.
Table 'Telemetry'. Segment reads 150, segment skipped 3.
Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

SQL Server Execution Times:
CPU time = 18062 ms,  elapsed time = 12791 ms.

And the results when executing against the same table where all deltastores have been compressed:

Table 'Telemetry'. Scan count 2, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 97544, lob physical reads 61, lob read-ahead reads 292647.
Table 'Telemetry'. Segment reads 259, segment skipped 6.
Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

 SQL Server Execution Times:
   CPU time = 641 ms,  elapsed time = 435 ms.

A tidy 30x reduction in CPU.

How to check if you're impacted

The dmv sys.column_store_row_groups tells you the status of each of your row groups - you're looking for any open row groups that are no longer being inserted into, which is extremely common if you partition by an ever-increasing datetime value, more so as you have smaller partitions.

select object_name(rg.object_id) as TableName
      ,rg.*
from  sys.column_store_row_groups as rg
where rg.state_desc = 'OPEN'

You can also use Niko's Columnstore Indexes Script Library (CISL) to get a wealth of information about your columnstore indexes.

If you have a lot of partitions some of these DMVs can be very slow - I'd suggest dumping them into a #temp table for querying/analysis.

Fixing the problem

The good news is fixing this is easy:

alter index CCS_Telemetry on dbo.Telemetry reorganize
with (compress_all_row_groups = on);

Although the command is easy, the impact to your system if you have a lot of open rowgroups might not be! You might instead opt to only compress a subset of partitions (much more typical if you have active partitions accepting inserts):

alter index CCS_Telemetry on dbo.Telemetry reorganize
partition = (1,2,3)
with (compress_all_row_groups = on);

The final clause (compress_all_row_groups) refers to the row groups in the partitions, not all row groups in the table.

More information about index maintenance is available in the MSDN article Columnstore indexes - defragmentation.

Keeping a work journal with VS Code

Timothy Addison — Wed, 28 Feb 2018 00:00:00 +0000

I've recently started to keep a work journal, and aside from being an incredibly useful document, it also gave me an excuse to learn a lot more about VS Code. After publishing an extension for VS Code yesterday, I'm now declaring my work journal flow done (at least for now...), and the rest of this post will walk you through my setup.

A sample journal entry

Workflow goals

In setting up a workflow there were a few requirements I had:

Easy to get up and running on a new machine
Easy to run on multiple machines (home/work/laptops)
Minimum possible friction to add something to the journal
Works in VS Code (my go-to editor for anything that doesn't merit VS2017 or SSMS)

The first two points boil down to sync, and crucially that sync should contain not only the journal itself but also the settings used to help achieve the third point (minimum friction edits) on every machine.

Keeping things in sync

The journal itself lives in a repository in Visual Studio Team Services, in which it is free to host unlimited repos for up to 5 users. Having access to a repo either by SSH (which I use on most permanent machines) or via a Microsoft login (which allows for quick in-browser edits in a pinch) is helpful. I'd have probably gone with GitHub if I had a plan which supported private repos.

In order to keep the VS Code configuration in sync I use the Settings Sync extension, which uses a private GitHub gist (which is free!) to sync settings between instances of VS Code. With this configured all extensions/user settings/keybindings/etc., are synced between all instances of VS Code. It's awesome.

Configuring the editor

Aesthetics aside (Solarized Light and the Fira Code font if you're curious), the most important actions I take on the journal are:

Pull from the repo
Add a timestamp
Add, Commit, and Push changes

The pull/push typically bookend my day with each machine I work at. Pulling from the repo is easy (builtin command), but adding a timestamp and the add/commit/push combination don't have an in-built solution.

For the timestamp I use the Insert Date String extension configured with the following custom format:

"insertDateString.format": "DDD MMM DD hh:mm:ss YYYY\n"

Binding this to a key means hitting the key inserts a timestamp and puts the cursor on a newline.

Solving for add/commit/push was actually much harder than I thought it would be, and after looking at a few extensions which came close (like Git Urgent! and Git Add & Commit) I ended up writing my own extension - GitLazy.

Although it was a fairly simple addon, it was my first extension and my first look at TypeScript, so I've learned a whole lot from that. In terms of time saved vs. time spent I think it should pay off in...a few years! My default commit message is configured as:

"gitlazy.commitMessage": "Update journal"

Putting it all together

One of the first things I do after logging in to a machine is now go to the second desktop (I'm using Windows 10), open my journal VS Code workspace, and put it into Zen mode (CTRL-K, Z by default). From that point on I'll flip back to that desktop (WinKey+Ctrl+Right) whenever I want to log something.

I use markdown by default, and most of the time log code in appropriately delineated blocks (as per the above screenshot), though I don't mind simply pasting something in if I want to keep it for reference.

Coming soon

Given the amount of time I spent optimising the flow of add/commit/push I find it highly unlikely I'll be able to leave the flow alone for too long. I've considered making the code workspace a startup task, or even pinning the workspace to the start menu for even faster switching (e.g. WinKey+1).

I'm debating right now whether or not to put some outputs from meetings in the journal or not - for 1:1s I typically keep notes organised per-person (in addition to a shared agenda) on something like OneNote or a word document in SharePoint. I'm now wondering if they should also get dumped into the log, or if perhaps migrating them to source-controlled markdown is a better format.

I'm also looking forward to writing something to parse out the journal, not to gain anything in particular from it right now, more because it'll be a chance to write some parsing code and see what can be done with the data...

Service Broker, Temporal Tables, and the 'Data modification failed' error

Timothy Addison — Sat, 17 Feb 2018 00:00:00 +0000

Temporal tables are a fantastic feature which we've enjoyed rolling out to replace some hand-rolled logging. Adding system versioning to a table has been mostly straightforward, though last week one of my colleagues saw some really odd behaviour that took the team a while to debug. Now we've understood the problem we're able to reproduce it 100% of the time (and subsequently come up with a workaround), though it definitely had us scratching our heads for a while - thanks for a super-interesting problem Ola!

We've previously had experience with highly concurrent modifications to a single row causing a data-modification error, and in every case we'd end up tracking down a bug which was causing unnecessary concurrent modifications.

Msg 13535, Level 16, State 0, Procedure HandleProcessPayment, Line 20 [Batch Start Line 0]
Data modification failed on system-versioned table 'TemporalBroker.dbo.Payment' because transaction time was earlier than period start time for affected records.

What had us really confused this time was that this was a very low-volume process, and we didn't observe any concurrency around the insert/update activity for the single row (confirmed with exhaustive XEvent-ing!).

The Setup

One of our applications uses service broker fairly heavily, the simplified flow looks something like this:

Web app inserts a payment record in state 'ready to pay'
Web app queues a message on service broker with the instruction to pay
Payments app is listening on that queue for instructions
Payments app receives the message in a transaction
Payments app requests a payment
Payments app updates the status of the payment to 'requested'
Payments app commits the transaction

After implementing system versioning on the table we started to see errors on the step which updated the payment.

The transactional flow (shown by time T, and session S - in this case session 1 is the payments app, session 2 is the web app) looks something like this:

T1, S1 - Begin Tran
T2, S1 - Waitfor receive...
T3, S2 - Insert Payment
T4, S2 - Queue PaymentProcess message
T4, S1 - Receive PaymentProcess message
T5, S1 - Update Payment - error

The clue was sitting in the error message - transaction time was earlier than period start time for affected records. At time T5 when the update happens the row is stamped with the time from T1, not T5. Because we use a transaction to pop messages and wait for one to arrive, we can end up processing the message at a timestamp before the Payment record is inserted. This behaviour is by design - and is called out pretty clearly in the official docs:

The times recorded in the system datetime2 columns are based on the begin time of the transaction itself. For example, all rows inserted within a single transaction will have the same UTC time recorded in the column corresponding to the start of the SYSTEM_TIME period.

As well as in the ISO technical report on SQL Support for Time-Related Information:

An UPDATE statement on a system-versioned table first inserts a copy of the old row with its system-time period end time set to the transaction timestamp, indicating that the row ceased to be current as of the transaction timestamp. It then updates the row while changing its system-period start time to the transaction timestamp, indicating that the updated row to be the current system row as of the transaction timestamp.

In all of our previous troubleshooting we were dealing with very short (typically implicit) transactions, and so we were incorrectly thinking about concurrency at the statement level, rather than the transaction level.

Repro

If you'd like to try this repro out yourself you can download this script to create the database and all objects required.

The two procedures we'll be looking at are the one which inserts the payment request:

create or alter procedure dbo.RequestPayment
as
begin
	insert into dbo.Payment
	( PaymentStateId )
	values
	( 1 );

	declare @paymentId int = scope_identity();
	declare @message xml =
            N'<ProcessPayment><PaymentId>'
            + cast( @paymentId as nvarchar(10) )
            + '</PaymentId></ProcessPayment>'
	declare @dialogId uniqueidentifier;

	begin dialog @dialogId
	from service PaymentProcessService
	to service 'PaymentProcessService'
	on contract PaymentContract
	with encryption = off;

	send on conversation @dialogId
	message type ProcessPayment
	( @message );
end

And the procedure which processes the payment:

create or alter procedure dbo.HandleProcessPayment
as
begin
	set nocount on;
	declare @message xml;

	begin tran;
		waitfor (
			receive top(1) @message = message_body
			from dbo.PaymentQueue
		), timeout 10000;

		if @message is null
		begin
			commit;
			return;
		end

		declare @paymentId int = @message.value('(/ProcessPayment/PaymentId)[1]','int');

		update dbo.Payment
			set PaymentStateId = 2
		where PaymentId = @paymentId;
	commit;
end

In this trivialised example the procedure does all the work, in the actual environment there was a single transaction and multiple commands executed (as well as the third-party calls to actually initiate a payment!).

To reproduce the error, execute dbo.HandleProcessPayment in one session (which will wait up to 10 seconds for a message to arrive), and then run dbo.RequestPayment in another session - you'll see the HandleProcessPayment procedure error out.

Workarounds

Depending on what you can change there are a few ways to work around this that can work:

Remove the timeout from the waitfor (if there is no sleep in the application you'll end up calling this a lot)
Remove the timeout from the waitfor and add a sleep (this increases average time to respond to a message)
Add a wait between inserting the Payment and sending the message (this also increases average time to respond)
Retry on receiving the error (in our case we hit this edge case 100% of the time on the first attempt to process)
Don't use system versioning (a trigger based solution with getutcdate() doesn't have this problem)

None of these are great options (in the end we lowered the waitfor timeout to a few hundred milliseconds and added the same delay before queueing the message - this eliminated all the errors), and we'll be considering what changes we could make to our messaging in the future to remove the requirement for a workaround. The current favoured option is a switch away from service broker where we don't need all the complexity and polling from the app.

Changing the owner of an SSRS subscription with PowerShell

Timothy Addison — Wed, 24 Jan 2018 00:00:00 +0000

If you're responsible for an SSRS instance you've probably got a script somewhere to handle changing the owner of a subscription (typically done when a user leaves/changes role and they've got subscriptions that need to keep running).

The official docs do provide a PowerShell script you can use to do this, though it is a little inefficient if you've got any latency between you and your instance and/or you have a lot of reports (it gets all reports, and then for each one gets all subscriptions).

SSRS exposes an API which lets you get all subscriptions in one go, which is much faster. The script below will take all subscriptions owned by Tim and assign them to Tom.

$oldOwner = "foocorp\tim.addison"
$newOwner = "foocorp\tom.addison"

$rs2010 = New-WebServiceProxy -Uri "http://localhost/ReportServer/ReportService2010.asmx" -Namespace SSRS.ReportingService2010 -UseDefaultCredential

$subsToMove = $rs2010.ListSubscriptions("/") | Where-Object { $_.Owner -like $oldOwner }

$subsToMove | ForEach-Object { $rs2010.ChangeSubscriptionOwner($_.SubscriptionID, $newOwner) }
$subsToMove | Select-Object Path, Report | Format-Table

You'll need to update the Uri in the above example if your report server is anywhere other than localhost.

The script will also output the list of subscriptions that have been modified.

Since late 2016 there has been a PowerShell module available that has started to provide a comprehensive set of SSRS administration functions, though at the time of writing it has the same items-then-subscriptions pattern which makes it much slower than the above script.

Finally, it's worth noting that if you only manage a small number of instances (or perhaps you're just more comfortable in T-SQL) there is a simple script you can use (originally posted at https://blogs.msdn.microsoft.com/miah/2008/07/10/tip-change-the-owner-of-sql-reporting-services-subscription/, which now 404s and I can't find the new home).

 DECLARE @OldUserID uniqueidentifier
 DECLARE @NewUserID uniqueidentifier
 SELECT @OldUserID =UserID FROM dbo.Users WHERE UserID ='foocorp\tim.addison'
 SELECT  @NewUserID=UserID FROM dbo.Users WHERE UserName = 'foocorp\tom.addison'
 UPDATE dbo.Subscriptions SET OwnerID = @NewUserID WHERE OwnerID = @OldUserID

Big thanks to Jim for nudging me in the PowerShell direction on this one - what a fantastic little cmdlet New-WebServiceProxy is!

SQLChecks v1 Released

Timothy Addison — Sun, 07 Jan 2018 00:00:00 +0000

After using SQLChecks to help tame our production instances for a few months, v1 has now been released. This first release includes documentation on all supported tests, as well as limited guidelines on how to structure any new tests/PowerShell functions.

Get the latest version of SQLChecks from GitHub.

The documentation in some cases includes suggestions for config values, and explanations on why you should care about testing for that particular value. These values and recommendations are drawn from a combination official sources (e.g. SQLRAP), books & blogs, as well as hard-won experience.

The goal with these tests is not just to identify areas where you're not meeting policy today, but to ensure that over time you don't have any accidental regressions (e.g. new databases, people changing configuration settings for testing and not putting them back...)

As an example consider the transaction log growth tests:

Max transaction log fixed growth

Reports on any database which has a fixed growth larger than the config value.

Transaction log with percentage growth

Reports on any database log with a percentage growth configured.

Even though you'll endeavour to design your environment to never need an auto-growth (by e.g. right sizing your databases, managing log backups, monitoring replica redo, managing long-running transactions), if your database log file does ever fill and triggers that growth you'll want it to complete quickly (as all log activity is suspended while the log grows). Because the log is zeroed out before use you could find yourself waiting a long time when your 1TB database log attempts to grow by 10%.

Having this check in place alongside existing measures you might have to manage log file growths (setting model appropriately for new databases, one-time reviews of all databases in violation, etc.) means that when something invariably slips through the cracks (or a DBA fat-fingers an update) you'll know about it and be able to take corrective action.

Visualising stored procedure call trees with SQLSpelunker

Timothy Addison — Sat, 23 Dec 2017 00:00:00 +0000

Whether debugging a problem in an existing system, or planning changes to an existing (complex) system, knowing how the call graph for a bit of SQL looks has been invaluable in quickly understanding the domain.

If you've worked in any system where a lot of the logic ends up in the database, you'll probably be nodding your head and remembering 'that' procedure which looked pretty simple to start with, and after following a few dependencies you were suddenly deep down the rabbit hole... (the longest I've looked at so far branches out into ~100 procedures).

Taking inspiration from What's in the box? Validating SQL Scripts with Powershell and Get Started with the ScriptDom I built SQLSpelunker to allow you to quickly go from a stored procedure name to a visual of the call tree for that procedure:

c:\SQLSpelunker>dotnet run "server=localhost;initial catalog=tempdb;integrated security=SSPI" "exec dbo.ProcOne;"

tempdb.dbo.ProcOne
-tempdb.dbo.ProcTwo
--tempdb.dbo.ProcThree

You can get started right away by grabbing the source from Github or download the latest build from the releases page. To build or run the code you'll need the .Net Core 2.0 SDK.

Read on for more details about what is currently supported, as well as details of how it works under the hood.

Features

Visualising simple call trees

Given the following schema:

use tempdb
go
create procedure dbo.ProcOne
as
exec dbo.ProcTwo;
go
create procedure dbo.ProcTwo
as
exec dbo.ProcThree;
go
create procedure dbo.ProcThree
as
select @@servername as ServerName;
go

Running SQLSpelunker against that database and the script exec dbo.ProcOne will generate the following output:

tempdb.dbo.ProcOne
-tempdb.dbo.ProcTwo
--tempdb.dbo.ProcThree

ProcOne calls ProcTwo calls ProcThree. Fairly straightforward.

Infinite loop detection

While walking the procedure tree if a procedure is encountered that exists anywhere as a parent, that branch stops at the repeat call. This will catch both simple cases (ProcA calls ProcA) as well as multi-step infinite loops (ProcA calls ProcB calls ProcA).

If we now add one more procedure to our schema:

create procedure dbo.CallsItself
as
exec dbo.CallsItself;
exec dbo.ProcOne;
go

Running SQLSpelunker against the same database with the script exec dbo.NeverCallsItself will return the following:

tempdb.dbo.CallsItself
-tempdb.dbo.CallsItself [*]
-tempdb.dbo.ProcOne
--tempdb.dbo.ProcTwo
---tempdb.dbo.ProcThree

The asterisk ([*]) signifies the procedure has previously been called by a parent procedure, and SQLSpelunker will stop trying to walk the call tree.

Default schema

If a procedure is executed without a schema name then it will be assumed that the default schema of the calling user is dbo. So the following statement:

exec proc;

When parsed result in:

tempdb.dbo.proc

Note that the database name is defaulted based on the connection string, or in the case of cross-database calls the 'current' database the procedure is executing in. If you do not specify a database name on the connection string SQLSpelunker will throw an error.

Cross-database calls

The procedure calls can span multiple databases. Given the following new procedures:

use tempdb
go
create procedure dbo.CrossDBCall
as
exec master.dbo.ILiveInMaster;
go
use master
go
create procedure dbo.ILiveInMaster;
as
exec tempdb.dbo.CrossDBCall
go

We'd see the following call chain:

tempdb.dbo.CrossDBCall
-master.dbo.ILiveInMaster
--tempdb.dbo.CrossDBCall [*]

Limitations

Exec @procedure

Only stored procedures executed by identifier are supported. If your statement contains any SQL in the following form it won't be reported:

declare @procName = 'dbo.WontBeSeen';
exec @procName;

Every procedure is output

The current implementation will report every procedure called in the code, even if that procedure isn't reachable (e.g. hidden behind an always-false statement).

create procedure dbo.ThisMaybeDoesNothing
as
if 1 = 0
begin
    /* Never reachable */
    exec dbo.ProcOne;
end

if rand() < 0.1
begin
    /* Sometimes reachable */
    exec dbo.ProcTwo;
end
go

This produces:

tempdb.dbo.ThisMaybeDoesNothing
-dbo.ProcOne
-dbo.ProcTwo

Missing procedures are still reported

When looking up the definition of a procedure if the definition can't be retrieved the call will still be reported. This is probably a feature more than a limitation as some procs which do exist won't have their definition reported (a good example being msdb.dbo.sp_send_dbmail). In the future 'missing' procs may get an annotation.

Usage

The command line arguments for the SQLSpelunker console app are:

A connection string to the initial database (which must contain a database)
The SQL fragment that should be parsed and walked

An example connection string that connects to tempdb on the localhost instance with windows authentication is:

server=localhost;initial catalog=tempdb;integrated security=SSPI

How it works

The flow of the program looks something like this:

Parse the SQL fragment and extract all stored procedure calls
For each stored procedure call get the definition if it isn't already in cache
For each stored procedure parse the definition and extract all stored procedure calls
Repeat until there are no more unique calls (e.g. also break on infinite loops)

SQL fragments are parsed using TransactSql ScriptDom - the official docs are a bit bare-bones, so I'd suggest starting with Ed Elliot's excellent blog category if you want to learn more.

Stored procedure definitions are extracted using the object_definition function.

The future of SQLSpelunker

I was pleasantly surprised at how little code was needed to solve the core problem of getting the definition and parsing all procedure calls.

The console app is helpful but not particularly easy to use (a quick-start with colleagues often took the form of 'download .net core, clone the repo, and then run this command - no there isn't just an exe you can download...'). A web version or powershell cmdlet are ideas I'm playing with to make it more accessible.

The other feature which would be great is 'what calls this procedure?'. Obviously that is a very different challenge to walking a tree (you're forced to go an get every procedure - potentially from every database! When you've got thousands of procedures that starts to take time, and keeping it snappy/interactive was a goal, and one of the reasons I didn't just recommend some of the existing tools out there for exploring schema).

Moving away from procedures only I'd love to extend this idea to report what tables/columns are being modified, though the way that gets visualised/interacted with is what I'm struggling with right now (is the question 'show me everything' or 'show me all procs in this chain which touch table foo/column bar?').

The longest/deepest proc I've seen so far

Each line below represents a nested call to another proc all branching out from a single call. Curious to know what else is out there :)

Auto deflating Event Hubs with a function app

Timothy Addison — Sun, 10 Dec 2017 00:00:00 +0000

EventHubs have supported auto-inflate/scale-up for a while now, but don't come with an equivalent to auto deflate/scale-down. If your workload doesn't have sustained throughput requirements you'll probably benefit from periodically scaling back.

Assuming you allow your hub to inflate to 10 throughput units but most of the time you only need 1, at current pricing that represents an overpayment of $0.27/hour, or $2,300/year. Over multiple hubs (don't forget your dev/test instances!) it quickly adds up.

Doing this manually is possible (and right now the comments on the auto-inflate article suggest this is the way to go), though we can build and deploy a simple function app to take care of all of our event hubs periodically. The great thing about auto-inflate is that if we do scale back and the workload needs more throughput, it'll scale right back up again.

Where is the auto deflate checkbox?

Pre-requisites

In order to deploy the solution we'll need:

Visual Studio 2017 with Azure Development workload (Community edition is fine)
A app service to deploy a function app to
A service principal which has owner permissions on the resource group(s) containing the namespaces we want to scale down
Details of your tenant, subscription, and resource group(s)

Function App

The Azure documentation will take you through creating a function app if you don't already have one (which will create the required app service).

Once the function app is created grab the values for AzureWebJobsStorage and AzureWebJobsDashboard from the application settings (they're both the same value). These are needed for you to test your function locally.

Service Principal

To create the service principal we're going to use Azure Powershell, though you can also use the Azure portal.

The script below has been adapted from the Azure authentication for dotnet SDK article.

We're granting ourselves the right to modify a single resource group in the example below - if you want the app to modify Event Hub namespaces in multiple resource groups you need to grant the 'Scaler' app Owner rights on each resource group

$subscriptionName = "<your subscription name>"
$appDisplayName = "Scaler" # This can be anything you want
$appPassword = "<a strong password for your app>"

$resourceGroupName = "<resource group containing your event hub>"

Login-AzureRmAccount # You'll be prompted to login
Select-AzureRmSubscription -SubscriptionName $subscriptionName

$sp = New-AzureRmADServicePrincipal -DisplayName $appDisplayName -Password $appPassword
New-AzureRmRoleAssignment -ServicePrincipalName $sp.ApplicationId -RoleDefinitionName Owner -ResourceGroupName $resourceGroupName

$sp | Select DisplayName, ApplicationId # You'll need the ApplicationId later

While you're connected you can use the following command to get the values you need for TenantId and SubscriptionId:

Get-AzureRmSubscription | Select-Object SubscriptionId, TenantId, SubscriptionName

Configuring the app for local testing

Clone the application from the ScaleDownEventHubs repository on GitHub.

Once cloned modify your local.settings.json file to look like the example file, using the values you acquired in the previous steps.

Your ClientId is the ApplicationId, and the ClientSecret is the password you used.

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "<your connection string>",
    "AzureWebJobsDashboard": "<your connection string>",
    "ClientId": "<your client id>",
    "ClientSecret": "<your client secret>",
    "TenantId": "<your tenant id>"
  }
}

You can now add one or more event hub namespaces to the function (in ScaleDown.cs).

var namespaces = new List<EventhubNamespace>
{
    new EventhubNamespace("<subscription id>", "<resource group>", "<namespace>", 1)
};

When the function runs it will use the credentials from your local.settings.json file to compare each namespace against the target capacity (the last argument - 1 in the example above), and if the namespace has a higher throughput it will reduce it.

In the portal you'll see reference to Throughput or Throughput Units - in the SDK this is captured by the Capacity property. For more information on the library we're using to manage the Event Hubs you can read the Event Hubs management libraries documentation.

To test this without making any changes you can comment out the update line, or set the capacity numbers to 20 (if the target capacity is higher than the current capacity it will not attempt to scale up - that is what auto-inflate is for!).

By default the function is set to run once a day at 01:30. For testing purposes you'll probably want to change it to run every minute.

// Default - daily at 01:30
public static void Run([TimerTrigger("0 30 1 * * *")]TimerInfo myTimer, TraceWriter log)

// For testing - every minute
public static void Run([TimerTrigger("* */1 * * * *")]TimerInfo myTimer, TraceWriter log)

For more examples of how to specify the frequency see the timer documentation.

If you run the example locally you should see output similar to the below, where we can see our namespace is currently at 5 Throughput units, so we take no action (as the target was 20).

Deploying the app

Publishing the app from Visual Studio will only deploy the function - by default the settings won't get deployed. You can either use the function CLI to deploy your settings, or add them in the portal. The portal will already have the storage account settings, so the key settings you need to add are ClientId, TenantId and ClientSecret.

Once published the app will start executing on the timer you have specified.

You can monitor execution through the function app logs, Application Insights (if you add the required integration), or the Azure activity log. Every time the capacity of the namespace is changed there will be entries in the operational logs.

Using the SQLChecks library for SQL Server configuration management

Timothy Addison — Sun, 03 Dec 2017 00:00:00 +0000

Getting started with configuration management can be pretty daunting. The number of frameworks/tools and their complexity can end up being a significant barrier to actually getting started (requiring you to answer both 'which framework should I use' and then 'how do I actually use it'). The key to making progress is to start small and make configuration management part of your daily/weekly checks.

To this end I built the SQLChecks library to make it easy to get started checking your SQL Server configuration, and start implementing the patterns that the bigger frameworks will require.

This post will walk through how to set up a basic configuration management process to record and check the state of your SQL Server instances using Pester and SQLChecks. Once built you can run it interactively (reporting to the console as shown below), or as part of a schedule to produce a report.

In this example we've failed one of the tests that is checking that the right trace flags are set on the server (and both of them are missing!)

Specifying your desired instance configuration

One of the key activities to onboard a server/service into configuration management is to specify what that configuration should look like. SQLChecks currently focuses on SQL instance configuration, and at the time of writing allows us to specify:

The value of any sp_configure configuration values
Any global trace flags (including checking that there are no global trace flags)
The number of SQL Server error logs

For each server you want to check you'll need a file that describes the instance configuration. Only the ServerInstance property is mandatory - everything else can be omitted, as only the listed properties will be checked. By convention this file should be named instance.config.json.

localhost.config.json

{
  "ServerInstance": "localhost",
  "SpConfig": {
    "MaxDegreeOfParallelism": 0,
    "XpCmdShellEnabled": 0,
    "ShowAdvancedOptions": 0
  },
  "NumErrorLogs": 14,
  "TraceFlags": [3226, 7412]
}

The SpConfig property is actually a collection of properties, and you can discover all the available options (as well as their current values) by executing the following command against one of your servers. This requires you have the DBATools module installed.

Get-DbaSpConfigure -SqlInstance localhost | Out-GridView

The ConfigName is what you need to use in your SpConfig section. For every value you want to check, list the ConfigName and value you expect.

You can put these text files anywhere, though I would suggest creating a private git repository for them. This will allow you to keep a history of your configuration changes, and require changes to be committed along with any explanations that may help later debugging (e.g. 'Adding TF3459 to benchmark redo performance for a week'). Changing the values first in source and then on the server is a great habit to get into (especially if you later leverage more fully-featured frameworks).

Once you have your config file(s) you're ready to see if reality matches the plan.

Installing SQLChecks

SQLChecks requires DBATools, so if you've not already you need to install that.

Once you've got DBATools you'll need to clone SQLChecks (you can also download a zip from Github). Once cloned you need to import the SQLChecks module.

c:\src>git clone https://github.com/taddison/SQLChecks.git
c:\src>Import-Module .\SQLChecks\src\SQLChecks -Force -Verbose

We're now ready to 'test' our configs - to make this easier I'd suggest creating a PowerShell script that will test one or more of your files. Examples of these can be found in the SQLChecks repo - included below is a file that will test a single config.

Note that we're referencing a folder we cloned as part of SQLChecks (c:\src\SQLChecks\tests), as well as a configuration file (localhost.config.json) - update the names/paths accordingly.

RunChecks.ps1

#Requires -Modules DBATools, SQLChecks

[string]$data = Get-Content -Path .\localhost.config.json -Raw
$data | ConvertFrom-Json -OutVariable configs | Out-Null

Invoke-Pester -Script @{Path='c:\src\SQLChecks\tests';Parameters= @{configs=$configs}}

You can then run the checks by executing the PowerShell file:

c:\src\SQLInstanceConfigs>.\RunChecks.ps1

If you've configured everything correctly you should see Pester start to run tests against your server.

What is happening

The high level flow for what happens when you execute RunChecks.ps1 is:

Checks to see if the DBATools and SQLChecks modules are available
Extract the contents of the specified file (localhost.config.json) to a variable called $data, using -Raw as we want it as a whole string rather than as a collection of lines (which we'd get by default)
Convert the $data string to Json, and store the result in a variable called $configs. We use Out-Null to suppress console output
Run Pester and complete all the tests (files with .tests.ps1 extensions) in the specified folder (c:\src\SQLChecks\tests), passing the $configs parameter to each test file

If we're executing tests against a folder full of config files the main change is that we're looping through files to extract configs (and then testing all of them), rather than testing a single file.

Right now there is only a single test file in the tests folder, which contains multiple tests for the SQL Instance. In the future you can expect to see files which perform tests against availability groups, databases, and Windows server.

Generating reports

In order to generate a report we're going to leverage Format-Pester to run all of our tests, and output them into a self-contained HTML report. This can then be saved for future reference or distributed via email/fileshare/etc.

You'll need to install both Format-Pester and PScribo (used by Format-Pester) to execute this script.

GenerateReport.ps1

#Requires -Modules DBATools, SQLChecks, Format-Pester, PScribo

$instances = Get-ChildItem -Path c:\src\SQLInstanceConfigs -Filter *.config.json
$configs = @()

foreach($instance in $instances) {
    [string]$configData = Get-Content -Path $instance.PSPath -Raw
    $configData | ConvertFrom-Json -OutVariable +configs
}
Invoke-Pester -Script @{Path='c:\src\SQLChecks\tests';Parameters= @{configs=$configs}} -PassThru | Format-Pester -Format HTML -Path c:\reports

This example pulls all configuration files from the specified folder (c:\src\SQLInstanceConfigs) and outputs the report to a target folder (c:\reports)

When configs don't match reality

A key part of configuration management is dealing with settings that don't match their configuration value. If you run your tests and discover that the MAXDOP setting isn't correct you might want that to be automatically handled. This isn't something that this solution handles (though you could easily extend it to do so!).

DSC has a very simple (yet powerful) model for thinking about configuration management that helps highlight the core difference between this solution and what I'd consider 'full' configuration management. Every resource in DSC (which could be something like an sp_configure value) supports three operations:

Get the value
Test the value against config
Set the value to something

You can see that we've got something that deals with the first two (get, test) only.

Though it might seem we're missing out on the most important part of configuration management by not implementing the 'set' functionality, that actually sits at the top of the pyramid. Knowing what your server configuration is supposed to look like and being able to verify it does or does not match are definitely where you should be starting.

Simplifying alert configuration in the OMS to Slack function app

Timothy Addison — Wed, 08 Nov 2017 00:00:00 +0000

In the last post we had a fairly complete (if limited) solution for the routing of arbitrary OMS metric alerts to Slack. After having used this in production for a while I can report that:

When configured correctly it works well (we get the right alerts in the right places)
Explaining how to configure a new alert is extraordinarily difficult
There is a seemingly arbitrary separation of alert configuration (some in the OMS webhook payload, some in the function app itself)

I was planning on extracting the configuration from the application, and rather than extract as-is I've made changes that will hopefully address the last two points.

Inspired by a recent Hanselman post I've opted to use CSVs and CSVHelper to store the alert configuration data. The tabular format and lack of references between config files leant itself to this task, and I wanted to start with a format that ensured it was easy to source control the configuration (so they can live and be versioned with the project).

If you want to look at the finished repo you can see it on GitHub - the rest of the post will focus on the key changes to configuring an alert, as well as how I've implemented those changes.

Configuring an alert

One of the most significant changes is the OMS payload. I've included the old & new payloads for a CPU alert below.

// Old
{
  "IncludeSearchResults": true,
  "WarningThreshold": 0.35,
  "CriticalThreshold": 0.45,
  "ValueMultiplier": 0.01,
  "Channel": "#oms-alerts",
  "LessThanThresholdIsBad": false,
  "AlertMessage": "Infra - CPU",
  "MetricName": "Processor Usage %",
  "FormatString": "P0",
  "ObservationThreshold": 3
}
// New
{
  "IncludeSearchResults": true,
  "MetricName": "Processor Usage %"
}

The OMS queries themselves remain unchanged.

The only additional piece of information we provide in the payload (IncludeSearchResults is needed to get OMS to send the metrics to our function) is the MetricName, and this acts as the key which maps to the configuration data in our app. The MetricName in the OMS payload needs to match the MetricName in all configuration CSVs.

The configuration data for each alert requires a minimum of a DefaultAlertConfig and a DefaultAlertNotificationConfig file. These are CSVs and are stored in the Configuration directory of the application (these must have their CopyToOutputDirectory set to CopyIfNewer to ensure they get deployed with the app).

The DefaultAlertConfig file contains the details needed to convert metrics from OMS (the ValueMultiplier), as well as the default alerting thresholds.

As a reminder the ValueMultiplier is needed to convert values which might be reported as whole numbers in OMS (e.g. 99% is reported as 99) into values appropriate for formatting (if we want to format as P0 we need the value to be 0.99). The value post-multiplication is also used when comparing to any alert thresholds.

One column which was significantly reworked is 'minimum violations to alert' - which is hopefully better named than its previous incarnation 'ObservationThreshold'. If in our example CSV below the processor query had 5 data points (e.g. per-minute over 5 minutes), at least 3 would need to be at or above 35% to trigger a warning alert (and 3 would need to be at or above 50% for critical).

defaultAlertConfig.csv

metricName,warningThreshold,criticalThreshold,lessThanThresholdIsBad,minimumViolationsToAlert,valueMultiplier
Processor Usage %,0.35,0.5,false,3,0.01
Free Space %,0.2,0.1,true,1,0.01
Free Megabytes,10000,5000,true,1,1

The DefaultAlertNotificationConfig file is used to determine how to format the alert, the message that should be used, and the default channel to route the alert to.

defaultAlertNotificationConfig.csv

metricName,channel,formatString,alertMessage
Processor Usage %,#alerts,P0,Infra - CPU
Free Space %,#alerts,P0,Infra - Drive
Free Megabytes,#alerts,N0,Infra - Memory

Customising alert thresholds or routing

Setting per-server rules for alerts is now done by adding records to the OverrideAlertConfig CSV file, which has the following format:

overrideAlertConfig.csv

metricName,machineName,warningThreshold,criticalThreshold,minimumViolationsToAlert
Processor Usage %,Server 2,0.3,0.4,

Metric and Machine are both mandatory, and every other field is optional. The way the above record reads is:

For Processor Usage % on Server 2
Override warning to 30%
Override critical to 40%
Inherit the minimum violations to alert from the default

For routing the configuration file is OverrideAlertNotificationConfig, and it allows us to override based on either metric, machine, or both.

overrideAlertNotificationConfig.csv

metricName,machineName,channel
,Server1,#Server1Team
Free Space %,,#memory-monitors
Processor Usage %,Server2,#server2-cpu

In this example you can see we have one rule that applies to a whole server, one that applies to a metric, and one which is targeted at a specific combination of a server and a metric.

The current behaviour for routing is that an exact match (metric + machine) will replace the default channel, and a partial match will add a channel to the list.

Reading configuration

The CSVHelper library was used to read CSV files, and this required minimal code. Using the out of the box defaults was all that was required to build the config population methods. The example below will read the CSV file using the headers (note these are case sensitive) and call the constructor to build each object.

// OverrideAlertNotificationConfig.cs
public class OverrideAlertNotificationConfig
{
    public OverrideAlertNotificationConfig(string metricName = null, string machineName = null, string channel = null)
    {
        MetricName = string.IsNullOrWhiteSpace(metricName) ? null : metricName;
        MachineName = string.IsNullOrWhiteSpace(machineName) ? null : machineName;
        Channel = channel;
    }

    public string MetricName { get; private set; }
    public string MachineName { get; private set; }
    public string Channel { get; private set; }
}

// ConfigHelper.cs
private List<OverrideAlertNotificationConfig> GetOverrideAlertNotificationConfigs()
{
    using (var tr = File.OpenText(_context.FunctionAppDirectory + "/Configuration/overrideAlertNotificationConfig.csv"))
    {
        var csv = new CsvReader(tr);
        return csv.GetRecords<OverrideAlertNotificationConfig>().ToList();
    }
}

The _context object is a Microsoft.Azure.WebJobs.ExecutionContext and is needed to get the root folder of our function app deployment (as we're deploying the configuration files along with the function app we need to get at them). More information can be found on the WebJobs Github wiki.

The defaults of CSVHelper automatically map an empty string in our CSV to null, so we don't have any additional configuration/maps required.

One area that does need some attention on the function app is reading/tolerating malformed CSVs. Right now if you get part of the CSV wrong, the whole thing fails.

Using configuration

As we now have the configuration isolated in a ConfigHelper class we're able to make the AlertProcessor a little simpler. The start of the ProcessAlert method (that deals with determining whether or not an alert sent from OMS should actually be considered an alert) now looks like this:

var alertConfig = _configHelper.GetAlertConfig(alert);

// Is this a < or > alert?
var comparison = alertConfig.LessThanThresholdIsBad ? LessThan : MoreThan;

// Aggregate metrics to produce a single summary record
var processedValues = alert.MetricValues.Select(mv => mv.Value * alertConfig.ValueMultiplier);

var totals = new
{
    Average = processedValues.Average()
    ,Min = processedValues.Min()
    ,Max = processedValues.Max()
    ,Critical = processedValues.Count(v => comparison(v, alertConfig.CriticalThreshold))
    ,Warning = processedValues.Count(v => comparison(v, alertConfig.WarningThreshold))
};

// Determine alert criticality
var isWarning = totals.Warning >= alertConfig.MinimumViolationsToAlert;
var isCritical = totals.Critical >= alertConfig.MinimumViolationsToAlert;

// If the alert doesn't cross the warning threshold return
if(!isWarning)
{
    return;
}

The final part of the method has also been simplified, so we're now getting closer to truly separating the alert logic from the notification logic:

// Where should the alert go
var notificationConfig = _configHelper.GetAlertNotificationConfig(alert);

// Build message
// ...elided

// Send message
foreach(var channel in notificationConfig.Channels)
{
    await SlackHelper.SendSlackMessage(channel, message);
}

Other changes

In order to keep the alerts as terse as possible stripping the domain name from a machine proved useful, so rather than seeing Server1.foo.corp was alerting we'd just see Server1. This is applied in function called by OMS, and so the configuration can refer to machines by hostname only.

machineName = machineName.Replace(".foo.corp", string.Empty).ToLower();

We also put the machine name into lowercase, and ensure when the configs are constructed that we convert all the servernames to lowercase too.

Getting funnel metrics right with Power BI when applying filters

Timothy Addison — Tue, 31 Oct 2017 00:00:00 +0000

In a previous post we built a simple set of recruitment funnel metrics that allowed us to plot the hiring funnel from a new application to (hopefully) a hire. The core measure we built was inclusive applications, which works fine until you start to filter your data, after which point you go from a set of data like this:

To a funnel that looks like this when filtered for the DBA role:

Which is kind of odd as we'd like to reflect the fact we phone screened two DBAs.

The reason the formula stops working when we filter on the role (DBA) the table we're working on only has 3 rows in, none of which have the Phone stage. Although it seems fairly contrived in our example dataset, once you start to slice the data on multiple dimensions it can be easy to end up in a place where you have nobody in a certain stage, particularly latter stages like Offer/Hire (for example: What did the funnel look like for DBA candidates in London got past the Phone Screen in Q1 17?).

The good news is this is an easy fix.

Fixing the funnel

Solving this issue is much easier if we first improve our data model, and take the 'Stage' dimension and create a dedicated dimension (or lookup) table for it. A great introduction to data modelling is available on PowerPivotPro.

Now we need to update our function to reference the dimension table (which as an added benefit simplifies the formula, as now we can use All(Table) rather than having to enumerate the columns):

Corrected Inclusive Applications =
CALCULATE (
    [Applications],
    FILTER (
        ALL (Stages),
        Stages[StageOrder] >= MAX ( Stages[StageOrder] )
    )
)

This gives us the chart we expect:

You can download a workbook with the data in here.

Troubleshooting an SSRS slowdown - portal delays, PREEMPTIVE_OS_LOOKUPACCOUNTSID, and bad plans

Timothy Addison — Sat, 21 Oct 2017 00:00:00 +0000

This tale takes place on an SSRS 2016 Enterprise instance running on Server 2012 R2.

The Report Server is slow

Users had reported that the SSRS instance was 'slow', and after opening the portal I could see what they meant. In addition to the home page taking a very long time to load (minutes), sometimes after loading I'd be presented with a menu bar and no folders/reports (despite knowing I should have seen a bunch of folders and reports).

The first port of call was the execution catalog, to see if any reports were running at all (as well as interactive rendering we have subscriptions and rendering happening via the API).

select      top 100 el.TimeStart
            ,el.ItemAction
            ,el.ItemPath
            ,el.Status
            ,el.*
from        ReportServer.dbo.ExecutionLog3 as el
order by    el.TimeStart desc

The query showed that subscriptions were working and some users were successfully browsing & running reports interactively. Knowing that the whole instance wasn't broken I was able to proceed to troubleshoot the specific issue of the portal misbehaving.

Looking into the portal issues

To summarise the problem after a few minutes of sense checks:

The portal is very slow to load (no errors observed in the browser console/no 404s/etc.)
Sometimes after the portal loads there are no reports/folders visible
Looking at user/folder settings (e.g. security) everything appears normal (permissions are intact, management interface works)
The report API is unaffected (can browse folders + render reports quickly and without any issues by navigating to /reportserver instead of /reports)
The servers hosting SSRS and the ReportServer database are not under high load (not capped on CPU/Memory/free drive space)
No configuration changes had been made recently

The portal looked to be the problem, so the next stop was the error logs. You can read more information about these logs online (Report Server Service Trace Log) - and in our case we're after the portal log file (Microsoft.ReportingServices.Portal.WebHost). Browsing these logs is always a little painful - someone somewhere has always left a misconfigured report, a report that belongs to someone who is no longer with the company - and shame on me we don't put these logs somewhere centrally queryable (and as such the errors only get dealt with when an end user reports an issue).

Ignoring all of the alerts which can get fixed later (subscriptions failing due to bad parameters, etc.), there were some unfamiliar alerts all of which have identical stack traces. The final error message in the trace was:

An error occurred within the report server database. This may be due to a connection failure, timeout or low disk condition within the database.

Which was immediately preceded by:

System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

This tells us our next stop should be the ReportServer database. The call stack of the error indicated it was part of GetFavoriteItems - which is why only the portal was having issues and not the API/reports themselves.

ReportServer database

Once we dropped into SQL the diagnosis was pretty easy.

exec sp_whoisactive @get_plans = 1;

A few queries running for over a minute, with a wait type of PREEMPTIVE_OS_LOOKUPACCOUNTSID. The query in question was dbo.GetAllFavoriteItems, and the offending statement was:

SELECT
    C.Type,
    C.PolicyID,
    SD.NtSecDescPrimary,
    C.Name,
    C.Path,
    C.ItemID,
    DATALENGTH( C.Content ) AS [Size],
    C.Description,
    C.CreationDate,
    C.ModifiedDate,
    SUSER_SNAME(CU.Sid),
    CU.[UserName],
    SUSER_SNAME(MU.Sid),
    MU.[UserName],
    C.MimeType,
    C.ExecutionTime,
    C.Hidden,
    C.SubType,
    C.ComponentID,
    CAST(1 AS bit)
FROM
   Catalog AS C
   INNER JOIN [dbo].[Catalog] AS P ON C.ParentID = P.ItemID
   INNER JOIN [dbo].[Users] AS CU ON C.CreatedByID = CU.UserID
   INNER JOIN [dbo].[Users] AS MU ON C.ModifiedByID = MU.UserID
   INNER JOIN [dbo].[Favorites] F ON C.ItemID = F.ItemID AND F.UserID = @UserID
   LEFT OUTER JOIN [dbo].[SecData] SD ON C.PolicyID = SD.PolicyID AND SD.AuthType = @AuthType

The wait type is coming from the calls into SUSER_SNAME. What this function does is takes the Sid and asks the local machine for the user name associated with it (by calling the LsaLookupSid function). If the mapping between Sid and user name isn't on the local machine, the machine will then ask the domain controller for the mapping.

Running the following query took over 2 minutes:

select suser_sname(u.Sid)
from dbo.Users as u

The only wait type observed was PREEMPTIVE_OS_LOOKUPACCOUNTSID, and each wait was between 70 and 150 milliseconds. In this case there are quite a few users that belong to a domain with a DC is located on a different continent to the report server (and the wait times correlate with the round-trip time from server to DC).

At this point I started looking at caching of the lookups (and it turns out there is a cache but only for 128 entries, and given we have 1000s of user records the cache will constantly get cycled if we are scanning the table). There is a registry setting to increase the cache size (after learning about the post from this SO question, this support article, and these docs), but before I went down that road I reflected that the DC hadn't moved overnight and the report server had been working fine up until today.

Plan regression

The problem (and the eventual solution) was the query plan. Looking again at the query it was clear we were pulling favourites for a single user, so we shouldn't be scanning the user table and looking up all the account name for everyone.

The current plan was scanning the user table twice, and for every row coming from the user table it was evaluating the suser_sname function for every row (represented by the compute scalars in the image below). The reason we scan twice is the query returns the name of the last modified by user as well as the created by user (for a report).

Looking in the table which holds user favourites I could see there was only a small number of rows, and so it seemed probable there was a better plan (especially as the portal had been performing well until recently, and there was no user who had favourited every report on the server…).

After running the procedure with a few users (including the user who had the most favourites) and getting only good (non-scan) plans I recompiled the procedure, and usual service was instantly returned (the query that had been running for minutes and timing out now finished in less than a second). The plan is as expected - get the favourites and then perform the Sid lookups only the users who own/modified one of the favourited reports.

What caused the plan to tip in the first place and when I don't know, as our ReportServer database didn't have one of the best features of 2016 enabled at the time (Query Store). That has now been rectified, and going forward the data about execution times/timeouts will now feed into our query store aggregation system and make spotting this kind of issue easy.

Bonus: Missing users

One additional issue we were facing is what happens when a Sid mapping isn't found. Rather than cache the fact there is no account returned for a given sid, nothing is cached and each request goes back to the DC. Some of our reports have been around for a long time and their their Sids are no longer in the directory - but SSRS still faithfully attempts to resolve the username of the report creator, sends the request on to the remote DC, and gets nothing back (the suser_sname function returns null).

This is easy to test - assume you've got two users (Tim and Tom). Tim still exists, but Tom doesn't. We're querying a remote DC for these users. To test yourself you'll need to get the Sid of a deleted user account - querying the Users table on an old ReportServer database might be the easiest way (the table also stores the account name, which is how SSRS is able to display information even if the Sid lookup returns null).

declare @timSid varbinary=0x00…
declare @tomSid varbinary=0x00…

select suser_sname(@timSid); -- takes 90ms
select suser_sname(@timSid); -- takes 0ms - cached

select suser_sname(@tomSid); -- takes 90ms
select suser_sname(@tomSid); -- takes 90ms - not cached

The Sid contains information about which domain the user belongs to - if you test with users removed from local domains you'll probably still be able to observe the result though the delta is much smaller (as LsaLookupSid is cached on the machine that makes the query - in this case our SQL Server).

Based on this, my recommendation is that you ensure the owner and 'lasted updated by' users of all reports are set to a user that currently exists, preferably in a local domain. Clearing old users out of the User table is something else you can do (so at least if the plan does tip you are minimising the impact).

Caching replica state to eliminate HADR_CLUSAPI_CALL waits on dm_hadr_availability_replica_states

Timothy Addison — Wed, 04 Oct 2017 00:00:00 +0000

We use tsqlscheduler to manage most of our SQL Jobs (a few hundred jobs in a several overlapping AGs), and when lots of schedules overlap (e.g. many concurrent jobs kick off on the hour) we saw waits and blocking on the function that was attempting to determine whether or not the server was the primary replica:

This function queries the DMV sys.dm_hadr_availability_replica_states, and we found that this DMV doesn't perform so well when the cluster size, number of AGs, and number of concurrent queries against the DMV rises.

Querying for the replica's role

select      @role = ars.role_desc
from        sys.dm_hadr_availability_replica_states ars
inner join  sys.availability_groups ag
on          ars.group_id = ag.group_id
where       ag.name = @availabilityGroupName
and         ars.is_local = 1;

Under the hood this DMV is calling into the Windows clustering APIs to determine the current status of the replica. There isn't a tonne of information out there - some comments on this SO answer confirm what the DMV is doing, and this MSFT post adds a little more detail. After checking some of the obvious items listed in the second post I decided that maybe we shouldn't be querying the DMV quite so often, and given how rarely the information changes ("is my node the primary replica?") caching this information seemed like a good fit.

Implementing caching

Caching of the replica state is available in any release of tsqlscheduler starting with 1.1. Each task now specifies whether or not it uses the cached replica check or not - most tasks should use the cached replica check, with the exception of the task which keeps the cache updated.

The caching is implemented by a task that persists all replica states for the given AG, and is queried by the GetCachedAvailabilityGroupRole function.

Update replica status

create or alter procedure scheduler.UpdateReplicaStatus
as
begin
    set nocount on;
    set xact_abort on;

    declare  @availabilityGroup nvarchar(128);

    select 	@availabilityGroup = ag.AvailabilityGroup
    from 	scheduler.GetAvailabilityGroup() as ag;

    merge scheduler.ReplicaStatus as rs
    using (
        select      ar.replica_server_name, ars.role_desc
        from        sys.dm_hadr_availability_replica_states ars
        inner join  sys.availability_groups ag
        on          ars.group_id = ag.group_id
        join        sys.availability_replicas as ar
        on          ar.replica_id = ars.replica_id
        where       ag.name = @availabilityGroup
    ) as src
    on src.replica_server_name = rs.HostName
    and	src.role_desc = @availabilityGroup
    when matched then
        update
            set rs.AvailabilityGroupRole = src.role_desc
    when not matched then
        insert ( AvailabilityGroup, HostName, AvailabilityGroupRole )
        values ( @availabilityGroup, src.replica_server_name, src.role_desc )
    when not matched by source
        then delete;
end

Get cached role

create or alter function scheduler.GetCachedAvailabilityGroupRole
(
	@availabilityGroupName nvarchar(128)
)
returns nvarchar(60)
as
begin
	declare @role nvarchar(60);

	select 	@role = rs.AvailabilityGroupRole
	from 	scheduler.ReplicaStatus as rs
	where	rs.AvailabilityGroup = @availabilityGroupName
	and	rs.HostName = host_name();

	return coalesce(@role, N'');
end

Deploying these changes eliminated the blocking behaviour, and we've seen the back of some fairly spectacular blocking chains we were able to produce when a server was under heavy load (yes, that is almost 3 minutes waiting to decide if the task can even run or not!).

Creating recruitment funnel metrics in Power BI

Timothy Addison — Sun, 10 Sep 2017 00:00:00 +0000

Creating a funnel chart in Power BI is extremely simple - drop your measure and group by column onto the canvas and format as a funnel chart. The tricky part is making sure your data is modelled correctly to give you the answers you need. If we assume that a simple 4-stage recruitment process (New Application - Phone Screen - Face to Face - Hire) has data captured in the following format:

When we create a funnel chart in Power BI we see the following:

That is technically correct (there are 3 applications in the New stage), but it isn't what we wanted (10 people applied and we hired 1 - which is not a 25% hire rate). While we could change our data (typically modifying the query to include multiple columns - one for each stage), this one is fairly easy to solve with a few DAX calculations to return the inclusive number of applications by stage.

The core measure we build returns the number of applications which have got to at least that stage. We do this by removing the filter context from both Stage and StageOrder (we typically visualise by Stage, but we rely on StageOrder for ranking stages), and then using the MAX function to pull the value of StageOrder from the filter context. In our case, the filter context will be the current group expression on the funnel.

Applications =
COUNTROWS( Applications )

Inclusive Applications =
CALCULATE (
    [Applications],
    FILTER (
        ALL ( Applications[Stage], Applications[StageOrder] ),
        Applications[StageOrder] >= MAX ( Applications[StageOrder] )
    )
)

Using this measure instead gives us the funnel we expected:

We can also define some additional DAX measures to report directly on the conversion rate between each step of the funnel. In the example below I've only defined a subset of the measures - typically you'd define these for all stages and conversion rates you cared about.

Hire =
CALCULATE (
    [Inclusive Applications], Applications[Stage] = "Hire"
)

Hire % All =
DIVIDE ( [Hire], [Applications], 0 )

Face to Face =
CALCULATE (
    [Inclusive Applications], Applications[Stage] = "Face to Face"
)

Hire % F2F =
DIVIDE ( [Hire], [Face to Face], 0 )

These can then be used when you want to compare those metrics between roles, locations, over time, or anything else you have in your model.

If you're currently recruiting and you don't know what your funnel looks like I encourage you to read how to recruit and then take another look at your funnel metrics (building them if needed).

You can download an example Power BI desktop file with the sample data and formulae here.

View currently executing tasks in tsqlScheduler

Timothy Addison — Sun, 03 Sep 2017 00:00:00 +0000

tsqlScheduler 1.0 was released today, and now contains a feature which allows you to view currently running tasks by storing information about that task in context_info.

The view scheduler.CurrentlyExecutingTasks can be queried and joined back to the core tables to produce results that tell us at a glance how long the task has been running for, and how long it took last time.

select	te.StartDateTime
		,datediff(second,te.StartDateTime, getutcdate()) as DurationSeconds
		,t.Identifier
		,lastResult.StartDateTime as LastStartTime
		,datediff(second,lastResult.StartDateTime, lastResult.EndDateTime) as LastDurationSeconds
		,lastResult.IsError as LastIsError
from	scheduler.CurrentlyExecutingTasks as cet
join    scheduler.GetInstanceId() as id
on      cet.Instanceid = id.Id
join	scheduler.Task as t
on		t.TaskId = cet.TaskId
join	scheduler.TaskExecution as te
on		te.ExecutionId = cet.ExecutionId
outer apply (
	select top 1 *
	from scheduler.TaskExecution as teh
	where teh.TaskId = t.TaskId
	and teh.ExecutionId <> te.ExecutionId
	order by ExecutionId desc
) as lastResult

In this example there are four tasks which have been running for 55 seconds. They normally take 60 seconds to run, and they last ran a minute ago (with no errors).

The rest of this post talks through how the mapping works.

Recap of tsqlScheduler

tsqlScheduler creates one SQL Agent job per entry in the scheduler.Task table. The agent job calls the scheduler.ExecuteTask procedure, which eventually calls sp_executesql to run the TSQL specified by the entry in the task table.

The execution of of each task is logged in the scheduler.TaskExecution table.

Storing information against each task

In order to tag each execution with the metadata we need to tie back to the task & execution data we use context_info. This allows us to store up to 128 bytes of data against a session which can then be queried through the dm_exec_requests and dm_exec_sessions DMVs.

The information we'll store against each execution are:

The scheduler instance identifier
The task id
The execution id

Each instance when deployed has a unique guid generated that identifies that instance (returned via the function scheduler.GetInstanceId). We need this as the task id is a number, and so for any SQL instance with multiple scheduler instances deployed (e.g. one standalone and multiple AGs) we use the instance id to disambiguate which instance the task belongs to.

Context info requires a binary payload, and rather than worrying about packing and unpacking a payload I've opted to use json to allow for a fairly flexible schema (I'd have preferred to use session_context but you can't query that from other sessions at the moment).

As each task is executed the ExecuteTask procedure calls into the following procedure to build and store our json data:

create or alter procedure scheduler.SetContextInfo
    @instanceIdentifier uniqueidentifier
    ,@taskId int
    ,@executionId int
as
begin
    declare @descriptor varchar(128) =
	    '{ "i":"' + cast(@instanceIdentifier as varchar(36))
	    + '","t":' + cast(@taskId as varchar(12))
	    + ',"e":' + cast(@executionId as varchar(12))
	    + '}';

    declare @binaryPayload varbinary(128) = cast(@descriptor as varbinary(128));
    set context_info @binarypayload;
end

SQL Agent does use pooled connections, though it resets each connection before each job is executed, which means that we don't have to worry about resetting the context_info after every execution.

Viewing task information

The view scheduler.CurrentlyExecutingTasks returns one row for each task currently executing, regardless of the scheduler instance the task was deployed into.

select  tasks.InstanceId
        ,tasks.TaskId
        ,tasks.ExecutionId
from    sys.dm_exec_requests as r
cross apply (
    select try_cast(r.context_info as varchar(128)) as ContextInfo
) as i
cross apply openjson (i.ContextInfo, N'$')
	with (
		InstanceId uniqueidentifier	N'$.i'
		,Taskid int	N'$.t'
		,ExecutionId int N'$.e'
	) as tasks
where r.context_info <> 0x
and   isjson(i.ContextInfo) = 1

Most sessions have no context_info associated with them (0x) so we can ignore those. For sessions which do have a payload we attempt to convert to varchar and then check if they're valid json. We then extract the json and correctly type the values.

To filter the currently executing tasks for the current scheduler instance only (the database that contains the view), you can modify the query to join onto the function that returns the instance's unique Id:

select cet.*
from   scheduler.CurrentlyExecutingTasks as cet
join   scheduler.GetInstanceId() as id
on     cet.Instanceid = id.Id

This is the same join that is used in the monitoring query at the start of the post, and is essential in deployments where multiple scheduler schemas are concurrently executing.

Building an OMS metric alert to Slack bridge with Azure functions

Timothy Addison — Tue, 29 Aug 2017 00:00:00 +0000

Previously we deployed a function app with three functions to send Slack notifications for OMS alerts based on CPU, memory, and disk. We'll now deploy a function with a single function designed to handle any OMS metric alert and send it to slack. As an added bonus we'll also support:

Overriding the thresholds to alert/critically alert based on server + metric
Use @channel to notify for critical alerts
Sending alerts to multiple channels based on either server or metric name
Configuring defaults directly in the OMS alert, allowing rapid deployment of new alerts

If you want to get started quickly:

You can grab a copy of the finished function app from this repo which you'll need to deploy as an Azure function (updating the Slack endpoint in SlackHelper.cs)
Configure your alerts using the examples shown at the bottom

Otherwise, read on for more details on what the function looks like, and more details on configuring the alerts and overrides.

Solution Overview

The solution is comprised of one or more alerts in OMS, an Azure function, and a Slack endpoint
When an alert is raised in OMS it calls our Azure function via a webhook and passes the alert configuration along with the alert search results that raised the alert
The Azure function uses the configuration and any deployed overrides to determine whether or not to alert, what channels to send the alert to, and whether or not to notify the channel (@channel)

Configuring the OMS alert

For every metric you want to alert on an OMS alert must be configured. While you can use almost any query you want to trigger an alert, there are a few things you'll have to do for each alert to get it to correctly route through the Azure function:

The alert must be a metric alert (trigger based on metric measurement rather than number of results)
The search query must only group by one value in addition to TimeGenerated (typically ComputerName, though in some cases you might have a Computer + InstanceName, in which case you should concatenate them with a pipe (Computer|InstanceName))
- If you group by more than one value the alert will not fire in OMS
The alert must specify a webhook
Has to specify a webhook action
The webhook must contain a custom JSON payload that contains all the parameters expected by the Azure function (see Custom JSON payload)

Example OMS Alert

Note that this example contains defaults appropriate for my environment - you may wish to modify the frequency of checks, thresholds, and suppressions.

Search Query

Perf
| where CounterName == "Available MBytes"
| summarize AggregatedValue= avg(CounterValue) by bin(TimeGenerated, 1m), Computer

Alert Settings

Time Window: 5 minutes
Alert Frequency: 5 minutes
Generate alert based on: Metric measurement
Aggregate value: Less than 10,000
Trigger alert based on: Total breaches greater than 3
Suppress alerts for: 20 minutes

Custom JSON payload

The custom JSON payload contains both generic information about the alert (how to format the numbers, what the alert is) as well as defaults for what constitutes a warning or critical alert.

{
  "IncludeSearchResults": true,
  "WarningThreshold": 10000,
  "CriticalThreshold": 5000,
  "ValueMultiplier": 1.0,
  "Channel": "#oms-alerts",
  "LessThanThresholdIsBad": true,
  "AlertMessage": "Infra - Memory",
  "MetricName": "Free Megabytes",
  "FormatString": "N0",
  "ObservationThreshold": 4
}

All parameters are mandatory (if you don't specify them the function will fail). Note that you cannot test the webhook by using the 'Test Webhook' button as it will not include the search results.

OMS Parameters

IncludeSearchResults: This must always be set to true so that the search query results will be included in the payload

Alert Information

LessThanThresholdIsBad: When true the aggregate value is compared against the warning/critical thresholds with a LessThan operator. When false GreaterThan is used.
ValueMultiplier: The value to multiply each metric value by before comparing to thresholds of formatting. Typically 1.0 or 0.01 (most perfmon counters report 50% as 50 rather than 0.5, which when formatted as P0 would show 5000%)
AlertMessage: The start of the message sent to the slack channel - see below for examples
FormatString: How the metric should be formatted, takes a .Net format string
ObservationThreshold: At least this many observations must be above (below) the threshold to trigger the alert (or upgrade from warning to critical)
MetricName: The name of the metric - this is included in the Slack notification

Alert Defaults

These can be overridden by the function based on the server/metric.

WarningThreshold: The value which if the metric is above (below) the function app won't send a Slack notification
CriticalThreshold: The value which if the metric is above (below) the function app will mark that notification as critical
Channel: The default channel for this alert. Channels can be added to but never removed from an alert.

WarningThreshold, LessThanThresholdIsBad will typically mirror the settings of the alert in OMS, while ObservationThreshold is typically one more than its corresponding OMS value.

Sample Messages

Assuming Server1 was low on memory, the warning & critical alerts based on the configuration above might look like this:

Infra - Memory [WARN] :: Server1 :: 9,400/9,500/9,600 (min/avg/max Free Megabytes)

Infra - Memory [CRIT] :: Server1 :: 3,400/3,500/3,600 (min/avg/max Free Megabytes

Sample OMS Alert Configs

CPU

Alerts when CPU exceeds a threshold.

Infra - CPU [CRIT] :: Server1 :: 56%/59%/61% (min/avg/max Processor Usage %)

Search Query

Perf
| where CounterName == "% Processor Time"
| summarize AggregatedValue = avg(CounterValue) by bin(TimeGenerated, 1m), Computer

JSON

{
  "IncludeSearchResults": true,
  "WarningThreshold": 0.35,
  "CriticalThreshold": 0.45,
  "ValueMultiplier": 0.01,
  "Channel": "#oms-alerts",
  "LessThanThresholdIsBad": false,
  "AlertMessage": "Infra - CPU",
  "MetricName": "Processor Usage %",
  "FormatString": "P0",
  "ObservationThreshold": 3
}

Memory

Alerts when free memory drops below a threshold. Measured in megabytes.

Infra - Memory [WARN] :: Server1 :: 9,400/9,500/9,600 (min/avg/max Free Megabytes)

Search Query

Perf
| where CounterName == "Available MBytes"
| summarize AggregatedValue= avg(CounterValue) by bin(TimeGenerated, 1m),Computer

JSON

{
  "IncludeSearchResults": true,
  "WarningThreshold": 10000,
  "CriticalThreshold": 5000,
  "ValueMultiplier": 1.0,
  "Channel": "#oms-alerts",
  "LessThanThresholdIsBad": true,
  "AlertMessage": "Infra - Memory",
  "MetricName": "Free Megabytes",
  "FormatString": "N0",
  "ObservationThreshold": 4
}

Disk

Alerts when a logical volume drops below a percentage threshold of free space. Note we group by a custom value (Computer + pipe + InstanceName).

Infra - Disk [WARN] :: Server1 - C: :: 5%/5%/5% (min/avg/max Free Space %)

Search Query

Perf
| where ObjectName == "LogicalDisk"
| where CounterName == "% Free Space" and InstanceName != "_Total"
| extend ServerDrive = strcat(Computer,"|",InstanceName)
| summarize AggregatedValue = min(CounterValue) by bin(TimeGenerated,5m), ServerDrive

JSON

{
  "IncludeSearchResults": true,
  "WarningThreshold": 0.2,
  "CriticalThreshold": 0.1,
  "ValueMultiplier": 0.01,
  "Channel": "#oms-alerts",
  "LessThanThresholdIsBad": true,
  "AlertMessage": "Infra - Disk",
  "MetricName": "Free Space %",
  "FormatString": "P0",
  "ObservationThreshold": 3
}

The Azure Function

The flow of a webhook through the Azure function is:

Webhook with alert configuration (custom JSON) and alert results (SearchResults) calls the OMSMetricToSlack function
OMSMetricToSlack parses the data into an Alert object and then passes that to the AlertProcessor
AlertProcessor determines the Computer name (we could be dealing with either a Computer or some combination of Computer + other value)
AlertProcessor looks up any threshold overrides for the combination of Computer + Metric (could be higher/lower)
AlertProcessor compares the metrics against the thresholds, and if ObservationThreshold number of values exceed threshold values the alert is set as warning or critical
- If the alert is neither a warning or a critical nothing further happens
AlertProcessor builds the list of channels to send to, starting with the default channel from the alert config and adding any additional channels from computer or metric name overrides
AlertProcessor builds a message and sends the message to every channel

OMSMetricToSlack

public class OMSMetricToSlack
{
    [FunctionName("OMSMetricToSlack")]
    public static async Task<object> Run([HttpTrigger(WebHookType = "genericJson")]HttpRequestMessage req, TraceWriter log)
    {
        var data = await req.Content.ReadAsAsync<OMSPayload>();

        var metrics = data.SearchResults.Tables[0].Rows.Select(r =>
        {
            return new MetricValue(DateTime.Parse(r[0].ToString()), Double.Parse(r[2].ToString()) * data.ValueMultiplier);
        });

        // Server1|E:
        var computerName = data.SearchResults.Tables[0].Rows[0][1].ToString();
        string instanceName = null;

        if(computerName.Contains('|'))
        {
            var split = computerName.Split('|');
            computerName = split[0];
            instanceName = split[1];
        }

        var alert = new Alert(
            data.Channel
            , data.WarningThreshold
            , data.CriticalThreshold
            , data.LessThanThresholdIsBad
            , data.AlertMessage
            , data.MetricName
            , data.FormatString
            , computerName
            , instanceName
            , data.ObservationThreshold
            , metrics
        );

        AlertProcessor.ProcessAlert(alert);

        return req.CreateResponse(HttpStatusCode.OK);
    }
}

AlertProcessor

The example alert processor class below contains a few overrides:

For the database servers, we want to warn when CPU exceeds 30%, and critically alert when CPU exceeds 50%
For the app servers, we want to warn when CPU exceeds 60%, and critically alert when CPU exceeds 85%
We want to send all database server alerts to the #database channel in addition to whatever the default channel is
We want to send all alerts based on SQL Batch Requests/second to the #database channel too

Note that the Computer should match exactly the name of the computer in OMS (case sensitive). The name of the performance counter is not the name of the performance counter in OMS, it is the value you've given to the MetricName property in the alert configuration. Take care to ensure you do not give duplicate metric names to different alerts.

public static class AlertProcessor
{
    private static Func<double, double, bool> LessThan = (double value, double threshold) => { return value < threshold; };
    private static Func<double, double, bool> MoreThan = (double value, double threshold) => { return value > threshold; };

    public static async void ProcessAlert(Alert alert)
    {
        // Is this a < or > alert?
        var comparison = alert.LessThanThresholdIsBad ? LessThan : MoreThan;

        // Machine-specific overrides?
        var (critical, warning) = GetMachineDefaultThresholdOverrides(alert.MachineName, alert.MetricName);
        var criticalThreshold = critical ?? alert.DefaultCriticalThreshold;
        var warningThreshold = warning ?? alert.DefaultWarningThreshold;

        // Where should the alert go
        IEnumerable<string> channels = new List<string>() { alert.DefaultChannel };
        channels = channels.Union(GetMachineChannels(alert.MachineName));
        channels = channels.Union(GetMetricChannels(alert.MetricName));

        // Aggregate metrics to produce a single summary record
        var totals = alert.MetricValues.GroupBy(_ => 1).Select(g => new
        {
            Average = g.Average(m => m.Value)
            ,Min = g.Min(m => m.Value)
            ,Max = g.Max(m => m.Value)
            ,Critical = g.Count(m => comparison(m.Value, criticalThreshold))
            ,Warning = g.Count(m => comparison(m.Value, warningThreshold))
        }).Single();

        // Determine alert criticality
        var isWarning = totals.Warning >= alert.ObservationThreshold;
        var isCritical = totals.Critical >= alert.ObservationThreshold;

        // If the alert doesn't cross the warning threshold return
        if(!isWarning)
        {
            return;
        }

        // Build message
        // Infra - CPU [CRIT] :: Server1 :: 56%/59%/61% (min/avg/max Processor Usage %)
        // Infra - Disk [WARN] :: Server1 - E: :: 5%/6%/6% (min/avg/max Free Space %)
        var instance = string.IsNullOrWhiteSpace(alert.InstanceName) ? "" : $" - {alert.InstanceName}";
        var message = $"{alert.DefaultAlertMessage} [{(isCritical ? "CRIT" : "WARN")}] :: {alert.MachineName}{instance} :: ";
        message += $"{totals.Min.ToString(alert.FormatString)}/{totals.Average.ToString(alert.FormatString)}/{totals.Max.ToString(alert.FormatString)} ";
        message += $"(min/avg/max {alert.MetricName})";

        if (isCritical)
        {
            message += " @channel";
        }

        // Send message
        foreach (var channel in channels)
        {
            await SlackHelper.SendSlackMessage(channel, message);
        }
    }

    private static (double? warningThreshold, double? criticalThreshold) GetMachineDefaultThresholdOverrides(string machineName, string metricName)
    {
        var combined = $"{machineName}|{metricName}";

        switch(combined)
        {
            case "DatabaseServer1|Processor Usage %":
            case "DatabaseServer2|Processor Usage %":
            case "DatabaseServer3|Processor Usage %":
                return (0.3, 0.5);
            case "AppServer1|Processor Usage %":
            case "AppServer2|Processor Usage %":
                return (0.6, 0.85);
            default:
                return (null, null);
        }
    }

    private static IEnumerable<string> GetMachineChannels(string machineName)
    {
        switch (machineName)
        {
            case "DatabaseServer1":
            case "DatabaseServer2":
            case "DatabaseServer3":
                return new List<string>() { "#database" };
            default:
                return new List<string>();
        }
    }

    private static IEnumerable<string> GetMetricChannels(string metricName)
    {
        switch (metricName)
        {
            case "SQL Batch Requests/sec":
                return new List<string>() { "#database" };
            default:
                return new List<string>();
        }
    }
}

SlackHelper

Ensure you replace the slackUri with your own Slack endpoint.

public static class SlackHelper
{
    public static async Task SendSlackMessage(string slackChannel, string message)
    {
        var slackUri = "https://requestb.in/1hoo74a1";
        var slackUsername = "Alerts";

        using (var client = new HttpClient())
        {
            System.Diagnostics.Debug.WriteLine($"{slackChannel} - {message}");

            var slackPayload = new SlackMessage() { Text = message, Channel = slackChannel, Username = slackUsername, LinkNames = true };
            var hook = new StringContent(JsonConvert.SerializeObject(slackPayload), Encoding.UTF8, "application/json");
            var response = await client.PostAsync(slackUri, hook);
        }
    }
}

Extending the solution

The solution is fairly robust and is fairly flexible in allowing you to alert on anything you can write a search query on. Some examples of things you might want to do with the function to help manage/debug/extend it are:

Add logging via calls to log.Info(...) or adding Application Insights to the function
Create additional test payloads to ensure your overrides are working as expected
Add some unit tests to verify the overrides
Support case insensitive, wildcard, regex, etc., overrides for computer name
Load configuration from a config file/Azure table storage
Add additional notification methods, potentially using Azure function integrations

One of the easiest ways to get alerting wrong is to configure yourself into never receiving alerts, so I would suggest before you go too crazy with your config you come up with some way of testing what you're coding in.

Happy alerting!

Migrating function app scripts to a class library

Timothy Addison — Wed, 23 Aug 2017 00:00:00 +0000

In the previous post we created a trio of function apps to send Slack notifications for OMS alerts based on CPU, Memory, and Disk. These were created using function app scripts, and the code was stored and tested directly in the Azure portal.

In this post we'll walk through the steps required to replace those scripts with a class library that exposes three functions that can be hosted by the Azure function app runtime. It will do exactly the same thing, but crucially will allow us to develop & test locally.

If you want to skp the interim steps and get straight to the finished code it is linked below. The code also contains the sample payloads, as well as a sample Pester test that validates the function completes without an error (returns a HTTP 200).

Migrating the function scripts

Create a new function app project (this step needs Visual Studio 2017 15.3). If you don't see the option to create function apps you might not have installed the Azure development workload. The official documentation walks through what is needed to build a function app with Visual Studio. (repo)

Now copy the script of the three functions into three separate files in the project folder. These are .csx (C# script) files in the portal, but we'll need to rename them to be .cs (C#) files. Right now the solution won't build, and we'll need to make a few changes to turn the C# scripts into valid C# files. (repo)

To get the solution to build:

Remove the first line of each script (#r "Newtonsoft.Json") - the Newtonsoft.Json package is already referenced by the Functions SDK, and references are managed at the project level rather than the file level
Wrap the Run method and supporting classes in a class (one class per file, use the name of the file so that the file CPUToSlack.cs would contain public class CPUToSlack)
Add usings for System.Threading.Tasks, System.Collections.Generic, and System.Linq, System.Net.Http, and Microsoft.Azure.WebJobs.Host

The project will now build. (repo)

If you run the project you'll see the function runtime start but an error message indicates that your app doesn't contain any functions.

To have our class library 'light up' as a function we need to tell the functions runtime which methods should be treated as 'functions'. We do this by specifying a FunctionName attribute on each method we wish to host as a function. We also need to provide some additional information about the input parameter (which for us is a webhook) so the function runtime knows what that function expects. Finally, we'll also need to add a using for Microsoft.Azure.WebJobs. (repo)

public class MemoryToSlack
{
    [FunctionName("MemoryToSlack")]
    public static async Task<object> Run([HttpTrigger(WebHookType = "genericJson")]HttpRequestMessage req, TraceWriter log)
    {

Running the function app again will show us that the runtime has discovered our functions and is now listening on a URL.

To test the functions I found it easiest to use the json payloads we created earlier and post them to the function using Powershell. The below code is a Pester test that will post the relevant payload to each function, and return success only if the function returns a HTTP 200 (OK) response code.

OMSToSlack.tests.ps1

$port = 7071
$uriBase = "http://localhost:$port/api"
$cpu = Get-Content cpu-payload.json
$memory = Get-Content memory-payload.json
$drive = Get-Content drive-payload.json

Describe "OMS to Slack Function" {
    It "Should return a 200 for a CPU payload" {
        (Invoke-WebRequest -Uri "$uriBase/CPUToSlack" -Body $cpu -Method Post -ContentType "text/json").StatusCode | Should Be 200
    }

    It "Should return a 200 for a Memory payload" {
        (Invoke-WebRequest -Uri "$uriBase/MemoryToSlack" -Body $memory -Method Post -ContentType "text/json").StatusCode | Should Be 200
    }

    It "Should return a 200 for a Drive payload" {
        (Invoke-WebRequest -Uri "$uriBase/DriveToSlack " -Body $drive -Method Post -ContentType "text/json").StatusCode | Should Be 200
    }
}

We can run the test by running the Invoke-Pester command in the folder that contains the file. A successful execution tells us that our functions are performing as we expect (or at least that they aren't throwing any exception!). As we make changes to the function code this test will allow us to verify that our functions still complete when we provide our known-good payloads. (repo)

The class library is now ready to be deployed and replace the script functions. To deploy and overwrite your functions you can download a publish profile from the Azure portal:

And then when you publish the project in Visual Studio select 'Import profile', and then Publish. If you browse to the Azure portal you'll notice that you can no longer see your function code (the .csx files), and instead you see the function.json file that points to your class library and entry point:

You can still test the function from the portal and watch the logs. You can also use the Pester test to run payloads against your live functions.

Next steps

As a final step I've made a few changes to the repo so that the current state no longer contains the duplicate class definitions in each of our functions (e.g. the OMSPayload class, the SlackMessage class).

We'll be making several changes to the functions next time to extract common functionality, promote some values to config, and then finally start extending the functions to provide the features I've talked about in previous posts.

Monitoring disk, CPU, and memory with OMS

Timothy Addison — Mon, 21 Aug 2017 00:00:00 +0000

In the previous post we built an OMS to Slack function app to alert on high CPU usage. We'll now extend that so we have alerts in place to cover us for the infrastructure basics - CPU, Memory and Drive Space.

The final code for the functions and example payloads are available in this GitHub repo.

OMS Alerts

These are designed to run over a 5 minute period, every 5 minutes, alerting on 3 consecutive breaches.

As we're building metric alerts OMS expects us to produce a single metric grouped by a single value - as such the Disk Space alert is grouping based on the server name concatenated with the drive letter. Grouping by ComputerName, InstanceName (which is what we're really interested in) will prevent the alert from firing altogether.

CPU Usage (%)

Perf
| where CounterName == "% Processor Time"
| summarize AggregatedValue = avg(CounterValue) by bin(TimeGenerated, 1m), Computer

Disk Space Available (%)

Perf
| where ObjectName == "LogicalDisk"
| where CounterName == "% Free Space" and InstanceName != "_Total"
| extend ServerDrive = strcat(Computer,"-",InstanceName)
| summarize AggregatedValue = min(CounterValue) by bin(TimeGenerated, 1m), ServerDrive

Memory Available (MB)

Perf
| where CounterName == "Available MBytes"
| summarize AggregatedValue= avg(CounterValue) by bin(TimeGenerated, 1m), Computer

Functions

To keep things simple I've built one function app per-alert, all following the basic pattern from the CPU alert built in the previous post. The table below summarises the alerts and links to the example payload generated by the OMS alert, as well as the text of the function.

Alert	Payload	Function	Measure	Warning	Critical
CPU	cpu-payload.json	CPUToSlack.csx	CPU Used %	75	90
Memory	memory-payload.json	MemoryToSlack.csx	Free Memory MB	10,000	5,000
Disk Space	drive-payload.json	DriveToSlack.csx	Free Space %	10	5

View all the above in a repo

If you choose to modify the thresholds then you'll need to ensure you modify them both in the OMS alert definition (under the 'Metric Measurement' section), as well as in the custom payload that specifies the warning/critical thresholds (under the 'Webhook' section).

Next steps

Now you've got monitoring in place for CPU, Memory and Disk for your whole estate you're probably interested in the fan-out/decoupling of trigger/response I mentioned in the previous post. We'll get there soon, though in the next post we'll look at migrating the script functions into a class library, and reducing a bit of duplication.

Analysing blob storage metrics with Power BI

Timothy Addison — Sat, 12 Aug 2017 00:00:00 +0000

Every new storage account has Storage Analytics enabled by default, which captures both logs and metrics relating to that storage account. The metric data is logged into a table in the storage account, which is used to power the metrics you see in the portal but can also be downloaded and used programmatically, by a tool like storage explorer, or with Power BI.

I recently found myself investigating why the cost of a resource group was so high (relative to expectations), and the driver turned out to be blob storage. As this application was supposed to be storing less than 1MB of data that was a surprise, and it was an even bigger surprise when the driver turned out to be the request charge.

In order to drill a level deeper than the billing API (which only reports requests/day), I built a Power BI template to understand what was happening to the storage account. This template can be used to analyse any storage account, and reports on capacity, requests by operation type, ingress and egress, as well as estimating costs.

You can download a copy of the template from this repo, or read on for more details about how to configure and use the report.

Using the Power BI template

To use the Power BI template you'll need Power BI Desktop, a copy of the template (latest version from this repo), and the storage account name and key.

When you open the template it will ask you to provide parameter values for the table endpoint. This typically takes the form https://storageaccountname.table.core.windows.net/ and can be found on the Properties blade of the storage account in the Azure portal. Note that even blob storage accounts have a table endpoint (the system uses table storage for all metrics, regardless of the kind of account).

Once you've provided the endpoint the template will connect and if this is the first time you've connected it will ask you for the account key. The account key can be found on the Access Keys blade of the storage account in the Azure portal.

The template will then download the capacity and metric data from the account. If you now save the Power BI file it will be saved as a report (.pbix), and next time you open this report you won't have to supply the details. In this way, you can quickly create reports for your key storage accounts that you can open and refresh as needed.

Date range

The report will pull all available capacity and metric data from the storage account. By default this is seven days - you may want to consider increasing this on your storage accounts as the cost is negligible if you are only storing metrics rather than logs - read more on storage metrics cost).

Both pages of the report are filtered to the last seven days by default - you can modify the date range using the filter at the top right of each page. If you want to perform more granular filtering (e.g. for a given period of time) you will need to filter using the DateTime column from the appropriate table (Blob Capacity or Blob Metrics).

Capacity information is captured once per-day, and the metric data we're using is captured once per-hour. More information is available about the tables we're reading from in the Metrics Table Schema documentation.

Capacity & Metrics page

The first page of the report shows an overview of capacity, requests by type, and ingress/egress over time. By default, the requests and egress data is shown by the hour. If your storage account is only infrequently used (e.g. once per day) then graphs that are rendered by the hour may look odd. Hovering over any graph that is per-hour and clicking the Drill Up button in the top left will change the graph from per-hour to per-day. The image below shows the same graph before and after drilling up.

This happens because storage analytics will only create an entry in the metrics table if something happens - if your account has no activity it won't create an entry and our metric is then not contiguous (you'll have a record at midnight and then nothing until the next day at midnight). There are workarounds for this (use a different visualisation, render the missing data as 0, create a custom date/time axis, etc.) but drilling up typically works well.

Estimated Costs page

The second page of the report allows you to quickly check what the storage account should be costing (either per-month in the case of capacity, or per-hour/day for requests). Note that this is an exploration tool only - your bill is the only source of truth for the actual cost!

The estimation page makes use of 'What-If' parameters. If you slide the request unit cost or storage cost counters the graphs will update to show the expected price. This is useful when evaluating the potential cost impact of switching storage types (e.g. LRS to GRS).

Building the report

Below are some details on how the report was built and how you can reconstruct some/all of the report.

Getting the data

Connecting to Table Storage (Get Data -> More -> Azure -> Azure Table Storage)
Selecting an existing table (the storage analytics tables are hidden - if you don't have any existing tables you'll need to create one)
Edit the query and then go to the advanced editor and replace the table name you selected with $MetricsHourPrimaryTransactionsBlob
Name this query Blob Metrics
Repeat the above but for the table $MetricsCapacityBlob and name the query Blob Capacity

Filtering out summary rows

You'll now have both datasets available. One thing to note is that the transaction metrics contains one row for every API call as well as a summary record (user;All and system;all). If you include these in any table/graph you'll be overstating your usage, so I suggest you either filter them out of the import, or you create a field which allows you to filter the summary rows out at the report level. To do this I split the RowKey column by delimiter (;) into two columns - Access Type (system/user) and Transaction Type (API called), and then created a custom column with the following DAX expression:

Transaction Type Group = IF([Transaction Type] = "all", "Summary", "Detail")

Applying a report filter to limit all rows to Detail then prevents any double-counting.

Date and time

To make working with dates easier (including correlating the capacity & transactions tables on a common date) the following steps should be taken when importing data:

Format the partition key as a date/time, and rename to DateTime
Create a copy of the formatted partition key column, format as a date, and rename to Date
Complete these steps for both imported tables

Once the data is loaded you'll want to create a custom calendar table with the DAX expression:

Date = CALENDARAUTO()

And then use the modelling view to create relationships between the capacity and metrics tables to the date table. You should then use this date on all date axes.

Expressions and formatting

By default, all the capacity and transaction metric data comes in as strings. During the import process, you will want to format these as whole numbers. Once imported creating and formatting measures will ensure that people are actually able to read your report (I don't know about you, but total egress of 3565504437 doesn't mean as much to me as 3.5GB). Some example measures I used:

Capacity B = SUM([Capacity])
Capacity MB = [Capacity B]/1024/1024
Capacity GB = [Capacity MB]/1024

For the costing data, my measures were all formatted as currency (USD) - you'll want to pick whatever currency your subscription is in, or use a decimal and ignore that.

What-if

To build the what-if functionality (e.g. cost per GB stored) you'll need to first create a what-if parameter. At the time of writing monthly blob storage costs range from $0.0224 to $0.061 per GB. To allow report users to explore this range we'll create a parameter that lets us go from 0.02 to 0.07 in 0.0001 increments (Modelling -> What If -> New Parameter):

Creating this will create a new table that has the series as well as the measure that returns the current value (based on any filters/slicers/calculation context). Note that the window is a shortcut to creating the table/expressions manually. Once we have the measure to compute cost we can then go on to create a measure to calculate the estimated monthly storage cost.

Storage Cost = GENERATESERIES(0.02, 0.07, 0.0001)
Storage Cost Value = SELECTEDVALUE('Storage Cost'[Storage Cost], 0.024)
Monthly Storage Cost = [Capacity GB]*[Storage Cost Value]

Parameterising the table endpoint

To parameterise the table endpoint a new parameter needs to be created in Edit Queries. Once created you can then edit the source step (double click the first step of each table import) and you'll be able to select from value or parameter:

You can manage the parameter value from the edit queries page (it will appear alongside tables and functions). When you export the report as a template it will then prompt for all parameter values when the template is first opened.

Browsing Azure Storage

For exploring Azure storage I highly recommend Storage Explorer, which will also show hidden tables and containers. The screenshot below shows an example of the $MetricsHourPrimaryTransactionsBlob table viewed with storage explorer.

The full text of the M queries in the template is included below. They both expect a parameter called TableEndpointURL.

M Queries

Blob Metrics

let
    Source = AzureStorage.Tables(TableEndpointURL),
    BlobMetrics = Source{[Name="$MetricsHourPrimaryTransactionsBlob"]}[Data],
    ExpandedBlobMetrics = Table.ExpandRecordColumn(BlobMetrics, "Content", {"TotalRequests", "TotalBillableRequests", "TotalIngress", "TotalEgress", "Availability", "AverageE2ELatency", "AverageServerLatency", "PercentSuccess", "PercentThrottlingError", "PercentTimeoutError", "PercentServerOtherError", "PercentClientOtherError", "PercentAuthorizationError", "PercentNetworkError", "Success", "AnonymousSuccess", "SASSuccess", "ThrottlingError", "AnonymousThrottlingError", "SASThrottlingError", "ClientTimeoutError", "AnonymousClientTimeoutError", "SASClientTimeoutError", "ServerTimeoutError", "AnonymousServerTimeoutError", "SASServerTimeoutError", "ClientOtherError", "AnonymousClientOtherError", "SASClientOtherError", "ServerOtherError", "AnonymousServerOtherError", "SASServerOtherError", "AuthorizationError", "AnonymousAuthorizationError", "SASAuthorizationError", "NetworkError", "AnonymousNetworkError", "SASNetworkError"}, {"TotalRequests", "TotalBillableRequests", "TotalIngress", "TotalEgress", "Availability", "AverageE2ELatency", "AverageServerLatency", "PercentSuccess", "PercentThrottlingError", "PercentTimeoutError", "PercentServerOtherError", "PercentClientOtherError", "PercentAuthorizationError", "PercentNetworkError", "Success", "AnonymousSuccess", "SASSuccess", "ThrottlingError", "AnonymousThrottlingError", "SASThrottlingError", "ClientTimeoutError", "AnonymousClientTimeoutError", "SASClientTimeoutError", "ServerTimeoutError", "AnonymousServerTimeoutError", "SASServerTimeoutError", "ClientOtherError", "AnonymousClientOtherError", "SASClientOtherError", "ServerOtherError", "AnonymousServerOtherError", "SASServerOtherError", "AuthorizationError", "AnonymousAuthorizationError", "SASAuthorizationError", "NetworkError", "AnonymousNetworkError", "SASNetworkError"}),
    ChangeInitialTypes = Table.TransformColumnTypes(ExpandedBlobMetrics,{ {"PartitionKey", type datetime}, {"TotalRequests", Int64.Type}, {"TotalBillableRequests", Int64.Type}, {"TotalIngress", Int64.Type}, {"TotalEgress", Int64.Type}, {"Availability", Int64.Type}, {"AverageE2ELatency", Int64.Type}, {"AverageServerLatency", Int64.Type}, {"PercentSuccess", Int64.Type}, {"PercentThrottlingError", Int64.Type}, {"PercentTimeoutError", Int64.Type}, {"PercentServerOtherError", Int64.Type}, {"PercentClientOtherError", Int64.Type}, {"PercentAuthorizationError", Int64.Type}, {"PercentNetworkError", Int64.Type}, {"Success", Int64.Type}, {"AnonymousSuccess", Int64.Type}, {"SASSuccess", Int64.Type}, {"ThrottlingError", Int64.Type}, {"AnonymousThrottlingError", Int64.Type}, {"SASThrottlingError", Int64.Type}, {"ClientTimeoutError", Int64.Type}, {"AnonymousClientTimeoutError", Int64.Type}, {"SASClientTimeoutError", Int64.Type}, {"ServerTimeoutError", Int64.Type}, {"AnonymousServerTimeoutError", Int64.Type}, {"SASServerTimeoutError", Int64.Type}, {"ClientOtherError", Int64.Type}, {"AnonymousClientOtherError", Int64.Type}, {"SASClientOtherError", Int64.Type}, {"ServerOtherError", Int64.Type}, {"AnonymousServerOtherError", Int64.Type}, {"SASServerOtherError", Int64.Type}, {"AuthorizationError", Int64.Type}, {"AnonymousAuthorizationError", Int64.Type}, {"SASAuthorizationError", Int64.Type}, {"NetworkError", Int64.Type}, {"AnonymousNetworkError", Int64.Type}, {"SASNetworkError", Int64.Type} }),
    DuplicatePartitionKey = Table.DuplicateColumn(ChangeInitialTypes, "PartitionKey", "Date"),
    SplitRowKey = Table.SplitColumn(DuplicatePartitionKey, "RowKey", Splitter.SplitTextByDelimiter(";", QuoteStyle.Csv), {"Access Type", "Transaction Type"}),
    ChangeFinalTypes = Table.TransformColumnTypes(SplitRowKey,{ {"Date", type date}, {"Access Type", type text}, {"Transaction Type", type text} }),
    RenamePartitionKey = Table.RenameColumns(ChangeFinalTypes,{ {"PartitionKey", "DateTime"} })
in
    RenamePartitionKey

Blob Capacity

let
    Source = AzureStorage.Tables(TableEndpointURL),
    BlobCapacity = Source{[Name="$MetricsCapacityBlob"]}[Data],
    BlobCapacityExpanded = Table.ExpandRecordColumn(BlobCapacity, "Content", {"Capacity", "ContainerCount", "ObjectCount"}, {"Capacity", "ContainerCount", "ObjectCount"}),
    ChangePartitionKeyType = Table.TransformColumnTypes(BlobCapacityExpanded,{ {"PartitionKey",type datetime} }),
    DuplicatePartitionKey = Table.DuplicateColumn(ChangePartitionKeyType, "PartitionKey", "Date"),
    RenamePartitionKey = Table.RenameColumns(DuplicatePartitionKey,{ {"PartitionKey", "DateTime"} }),
    ChangeTypes = Table.TransformColumnTypes(RenamePartitionKey,{ {"Date", type date}, {"Capacity", Int64.Type}, {"ContainerCount", Int64.Type}, {"ObjectCount", Int64.Type} })
in
    ChangeTypes

Updates and suggestions

If you have any suggestions for the report (either formatting, measures, default reports) please let me know.

The version of the template used when building this blog post can be downloaded here (the version on github may have since been updated).

Building better OMS alerts with function apps

Timothy Addison — Sun, 06 Aug 2017 00:00:00 +0000

OMS (Operations Management Suite) allows you to easily build infrastructure alerts to tell you when a server is using a lot of CPU, low on disk space, etc., but if you've tried to use the alerts to trigger webhooks to something like Slack you've probably come across a pretty painful limitation - getting the name of the computer that triggered the alert can't be done dynamically, meaning you either have to settle for an alert which tells you "One of your computers is using a lot of CPU!", or create one alert per-computer, and hardcode the computer name into the alert.

We'll use a simple function app to not only report which server triggered the alert, but have far more granular control over what we report in Slack:

If you want to skip straight to the final code it is available in this GitHub repo.

Pre-requisites

An OMS account (info)
At least one machine reporting % Processor Usage to OMS
An Azure function app (info)
A Slack account with an incoming webhook endpoint configured (info)

Getting all these configured is beyond the scope of this blog post.

Configuring the OMS alert

We're going to configure an OMS alert to fire when CPU utilisation is high. The values I'm using for thresholds/timings are useful in my current environment and might not be right for you - modify as required. The query we'll base our alert off is:

Perf
| where CounterName == "% Processor Time"
| summarize AggregatedValue = avg(CounterValue) by bin(TimeGenerated, 1m), Computer

If you look at this query in Log Analytics you'll see it returns the per-minute CPU utilisation for every machine linked to your OMS account. We're going to set our alert to evaluate this query every 5 minutes, and look at the prior 5 minutes of data. In our environment we want to know any time a single computer exceeds our threshold for CPU % for 3 consecutive minutes. The thresholds we've established to trigger a warning is 75%. We also want to suppress any alerts following a trigger for 20 minutes (to prevent spam). In the OMS alert editor that looks like this:

The way alerting & suppression works is per-machine. This means that if two different machines are see-sawing around the threshold such that one of them is always above and the other is always below, this alert will not fire. This is the behaviour we want - only alert us when a single machine stays above the threshold for 3 measurement periods. Make sure you take your measurement period into account when building alerts - if you measure CPU every 5 minutes only it won't make any sense to look at per-minute granularity.

The final part of configuring the OMS alert is to configure the outbound webhook that OMS will call every time the alert fires. Note that if multiple machines trigger the alert in a single evaluation period, OMS will trigger the alert once for every machine.

While testing I suggest you use RequestBin to capture the data that OMS sends. When configuring the webhook ensure you check the include custom JSON payload option, and add the following JSON:

{
  "IncludeSearchResults": true,
  "WarningThreshold": 75,
  "CriticalThreshold": 90
}

The IncludeSearchResults tells OMS that we want the search results generated to be included in the webhook payload. OMS will only include the search results for the machine which has triggered the alert. You can see an example of the payload generated in this payload file.

The warning & critical threshold values are provided for use in function app, and we'll use them to allow us to have a single OMS alert for both warning & critical, rather than having to create two alerts.

When correctly configured your OMS webhook config will look like this (though with your only RequestBin URL/function app URL):

You can test the webhook, though OMS will not run the query and include the search results. If you want to see what the search results look like you'll need to let the alert fire - set your threshold to something low (e.g. 1%) and the alert will fire within a few minutes.

Building the Function app

Create a new function app using C# Webhook template. The template also creates HTTP output, though we're not going to use that so you can delete it, leaving your function app with a HTTP trigger only.

We'll create a few helper classes to work with the incoming data from OMS, as well as create our Slack webhook. To build the OMS classes I used json2csharp.com, and pasted into the JSON payload from OMS. This generates a pretty verbose class, which you can strip back to the below.

public class OMSPayload
{
    public int WarningThreshold { get; set; }
    public int CriticalThreshold { get; set; }
    public SearchResults SearchResults { get; set; }
}

public class SearchResults
{
    public List<Table> Tables { get; set; }
}

public class Table
{
    public string TableName { get; set; }
    public List<Column> Columns { get; set; }
    public List<List<object>> Rows { get; set; }
}

public class Column
{
    public string ColumnName { get; set; }
    public string DataType { get; set; }
    public string ColumnType { get; set; }
}

In order to send a message to Slack we'll use a fairly simple class:

public class SlackMessage
{
    public string text { get; set; }
    public string channel { get; set; }
    public string username { get; set; }
}

The function app is going to:

Parse the OMS JSON into the classes above
Determine if we're in a warning or critical state
Extract the name of the computer alerting
Build a message with the computer name and some simple metrics
Send the message to Slack

The full text of the function (assuming the default input name of req) is below. Note this excludes the class definitions above, which would normally go at the bottom of the file.

#r "Newtonsoft.Json"

using System;
using System.Net;
using System.Collections;
using Newtonsoft.Json;
using System.Text;

public static async Task<object> Run(HttpRequestMessage req, TraceWriter log)
{
    var slackUri = "https://requestb.in/su78d1su";
    var slackChannel = "#webhook-tests";
    var slackUsername = "OMS";
    var criticalMeasurementCount = 3;

    var data = await req.Content.ReadAsAsync<OMSPayload>();

    var warningThreshold = data?.WarningThreshold ?? 75;
    var criticalThreshold = data?.CriticalThreshold ?? 90;

    var aggregatedResults = data.SearchResults.Tables[0].Rows.GroupBy(r => r[1].ToString())
                            .Select(g => new {  Computer = g.Key
                                                ,Average = g.Average(r => Double.Parse(r[2].ToString()))
                                                ,Warning = g.Count(r =>  Double.Parse(r[2].ToString()) >= warningThreshold)
                                                ,Critical = g.Count(r =>  Double.Parse(r[2].ToString()) >= criticalThreshold) });

    var message = string.Empty;
    var critical = false;

    foreach(var result in aggregatedResults)
    {
        message +=  $" - {result.Computer}:[{result.Warning} >{warningThreshold}% | {result.Critical} >{criticalThreshold}% | {(result.Average/100):P0} avg]";

        if(result.Critical >= criticalMeasurementCount)
        {
            critical = true;
        }
    }

    message = $"Infra - CPU {(critical ? "Critical" : "Warning")}" + message;

    using(var client = new HttpClient())
    {
        var payload = new SlackMessage() { text = message, channel = slackChannel, username = slackUsername };
        var hook = new StringContent(JsonConvert.SerializeObject(payload), Encoding.UTF8, "application/json");
        var response = await client.PostAsync(slackUri, hook);
    }

    return req.CreateResponse(HttpStatusCode.OK);
}

If all of the code and class definitions are present in the same file in the function app (run.csx) you should now be able to test this with the sample payload from OMS, or point OMS directly at the function.

The format of the message produced is:

Infra - CPU Severity - ServerName:[WarnCount >Warn% | CritCount >Crit% | Mean % avg]

Where WarnCount is the number of times in the sampling period that the metric exceeded Warn, and CritCount the same for Crit. In the below example message we can see the severity is critical, and that 5/5 measures exceeded the warning threshold of 60%, and 5/5 measures exceeded the critical threshold of 90%. This server doesn't look too healthy!

Infra - CPU Critical - Server1:[5 >60% | 5 >90% | 100 % avg]

You'll notice the code also uses a RequestBin for testing the Slack alerts. When you're done testing both the OMS webhook URL and the Uri specified in the Function app should point to their live endpoints.

Next steps

As configured, the solution will now start routing Slack alerts to your chosen channel, enriched with information from the search results included with each alert. In the future we'll extend the solution to support fan-out (1 alert notifying multiple channels), notifications (conditional @channel to get attention), and decoupling the notification from the trigger.

Updated 2017-09-08 for new OMS query language/search payload.

Using the SQL Infrastructure Tests repo in your environment

Timothy Addison — Sun, 09 Jul 2017 00:00:00 +0000

Update 2017-03-12: The SQLChecks library builds on the ideas in this post and delivers an upgraded version of SQLInfrastructureTests. This blog post discusses how to use the library.

The SQL Infrastructure Tests repo is generic and knows nothing about your infrastructure. This makes it very easy to clone and use anywhere, but also means you need to provide some configuration information to actually make it useful in your environment. Typically you want that environment specific information to also be in source control, you might be tempted to create a fork of the repo and make customisations for your environment.

This nearly always ends in pain when you need to pull updates from the repository. A better pattern is to create helper repositories (which would be privately version controlled). These are where you make all your environment specific changes (your server names, etc.). This has the added advantage of making it much harder to accidentally publish environment specific information to GitHub (say if you were to accidentally create a PR of your private fork…).

The public repo we're consuming is SQLInfrastructureTests, and the private repository that we'll be using is SQLInfrastructureTests-Helper.

The Install.ps1 script lives in our helper and will either clone the repository (if it doesn't exist), or update it if does.

Install.ps1

$repoName = "SQLInfrastructureTests"
$repoUri = "https://github.com/taddison/$repoName.git"

if(!(Test-Path -path "../$repoName")) {
    git clone $repoUri "../$repoName"
} else {
    Set-Location -Path "../$repoName"
    git pull
}

The RunTests.ps1 script then executes the pester tests that exist in the public repository. Because this file lives in our private repository, you would be able to use your own server names (or pull them from a CMS), and optionally specify subsets of tests to execute against different servers.

RunTests.ps1

$servers = @('localhost','.')
$repoName = "SQLInfrastructureTests"

# Run all the tests
Invoke-Pester -Script @{Path="../$repoName";Parameters= @{servers=$servers}}

# Run a single test
Invoke-Pester -Script @{Path="../$repoName";Parameters= @{servers=$servers}} `
              -TestName "*SQL Agent*"

The final line of the script above shows how you can use the TestName parameter of Invoke-Pester to limit the tests which are executed. Pester also supports tagging your tests, and this can be combined with name filtering to give your even more fine grained control over what you want to execute.

The idea of using helper repos can be applied fairly generally, and is useful anywhere you want to consume an open source project (and get the latest updates easily). I've also used this technique to have a single 'SQL Tools' repo in version control, which any developer can check out and run the install script on to get the latest version of SQL scripts & tools like the First Responder Toolkit.

Checking SQL Agent job ownership with Pester

Timothy Addison — Sat, 22 Apr 2017 00:00:00 +0000

Ensuring your jobs are all owned by SA is a best practice I've used to help minimise the chance of an SA job not running correctly due to the owners login being disabled, or there being an issue with authentication.

There are plenty of ways of going about this, though the most flexible I've found so far is making it an automated infrastructure test with Pester. If you're not familiar with Pester I encourage you to check out the Pester Github site, and then for more SQL specific details browse the Pester category of posts by SQL DBA With a Beard.

Once Pester is configured, the simplest way to execute a test is to run the following.

$query = "select count(*) as Jobs from msdb.dbo.sysjobs as j where j.owner_sid <> 0x01"

Describe "SQL Agent on localhost" {
    It "has jobs only owned by sa" {
        (Invoke-Sqlcmd -ServerInstance localhost -Query $query).Jobs | Should Be 0
    }
}

This executes the query against our localhost, and compares the result (count(*) aliased as Jobs) against the expected value of 0. When it works we get the following output:

If there are any jobs which are not owned by SA then we'll be told how many there are.

To really leverage this you'll want to target multiple servers. To do that from a single script we can provide a hard-coded list of servers and then call the test in a loop.

$query = "select count(*) as Jobs from msdb.dbo.sysjobs as j where j.owner_sid <> 0x01"
$servers = @('localhost','.')

foreach($server in $servers) {
    Describe "SQL Agent on $server" {
        It "has jobs only owned by sa" {
            (Invoke-Sqlcmd -ServerInstance localhost -Query $query).Jobs | Should Be 0
        }
    }
}

The list of servers could come from a text file, or a CMS Server.

Moving from a single script to a solution

As you go on to write more tests you will probably want to take advantage of Pester's ability to isolate tests in test files (with the .tests.ps1 extension) and want to run them all. Typically you'll want to manage the list of servers you execute the test against outside of the test itself. What I suggest is making the server list a common parameter for all your SQL test files, and then populating the server list in a 'runner' class.

In the below example you'll see we connect to a CMS to pull the list of servers to execute against (based on a group called 'Instances to Test'). The query has also been modified to ignore jobs in the 'Report Server' category, which is what SSRS uses when it creates jobs to support subscriptions (none of which are ever owned by SA).

RunTests.ps1

$cms = "localhost\CMS"

$servers = invoke-sqlcmd -Server $cms -Query "
SELECT ssrsi.server_name
from msdb.dbo.sysmanagement_shared_registered_servers_internal as ssrsi
join msdb.dbo.sysmanagement_shared_server_groups_internal as sssgi
on sssgi.server_group_id = ssrsi.server_group_id
where sssgi.name = 'Instances to Test'" | Select-Object -ExpandProperty server_name

invoke-pester -Script @{Path='.';Parameters= @{serverList=$servers}}

SQLAgent.tests.ps1

Param(
    [object]$servers
)

$query = "
select count(*) as Jobs
from msdb.dbo.sysjobs as j
join msdb.dbo.syscategories as c
on c.category_id = j.category_id
where j.owner_sid <> 0x01
and c.Name <> 'Report Server'"

foreach($server in $servers) {
    Describe "SQL Agent on $server" {
        It "has jobs only owned by sa" {
            (Invoke-Sqlcmd -ServerInstance $server -Query $query).Jobs | Should Be 0
        }
    }
}

As you create additional infrastructure tests you can drop them in the folder and Pester will automatically run them on your list of target servers. I've got a repository which contains some example tests on Github.

Note that the test we have coded for looks to ensure all jobs are owned by SA. To workaround the issues described at the start of the post (e.g. can't authenticate a user account to run a job) we should say that user accounts shouldn't own jobs. In my current environment that translates to all jobs must be owned by SA. In your environment you might have a dedicated SQL account for jobs (or something similar), and so you may wish to modify this check.

Additional Reading

sp_Blitz One of the checks sp_Blitz currently implements is Jobs Owned by User Accounts. If you're running this regularly it'll help catch the SQL Agent job issue, as well as many others.

Format-Pester Take the output of Invoke-Pester and produce self-contained HTML summary pages.

tSQLScheduler Github project used to create agent jobs automatically from data, ensuring they're always created as SA.

What makes an excellent database engineer?

Timothy Addison — Sun, 09 Oct 2016 00:00:00 +0000

The question of what makes an excellent database engineer is one I'm always asking myself and increasingly find myself answering for others. The question typically isn't phrased quite so directly, and instead shows up in a variety of discussions:

In 1:1s this might be part of a discussion on long-term career goals and progress against them
During the recruiting process some of the best candidates will probe to understand how I define greatness in that role
When working with other leads we want to try and understand why certain team members are great, and look to see what behaviours we want to encourage elsewhere

While the specifics will always vary with the individual, the team, and the platform, I believe that the below fairly accurately captures what I currently think of as the key traits & beliefs an excellent database engineer should possess:

Deep technical mastery of one or more areas
Broad knowledge of the platform and the ecosystem it works in
Ownership of problems & their solutions end to end
Does not tolerate badness
Treats the system as a customer

This is largely based on the role of a database engineer who is both a developer, an administrator, and an operations guy (outside of some specialist exceptions I don't think the dedicated role of DBA really has a future - perhaps a subject for another post). I primarily work in the Microsoft ecosystem but I believe the detail below applies for any ecosystem, and probably more generally for any software engineering role.

Each item is discussed in more detail below.

Deep technical mastery of one or more areas

A whole heap of other qualities are rolled up into this goal. The fact this goal is never actually finished (can anyone ever say they're finished learning about query optimisation? How long would that statement be valid for?) means you need to be both humble in accepting you have something to learn, and curious enough to keep you digging into that area.

Each additional level of mastery means you're now able to teach that area to other people (teaching/speaking/writing about an area at level N requires level N+1 knowledge), that you're able to work in that area with reduced effort and increased confidence, and that any project which interacts with that area will benefit from your input.

Database engineering excites me so much as it exposes a huge surface area for learning - you're at the confluence of academic research, hardware & software advances, product advances, as well as the disciplines of programming, systems architecture, and system administration. On top of all that sits the solution you'll build to solve a real problem, and knowing the foundations of what you build is going to help you deliver a better project (cost, performance, reliability).

Broad knowledge of the platform and the ecosystem it works in

You can't solve a problem using a tool you don't even know exists. That's a rough analogy but it holds even when you start to stretch that up to the system level. Consider the following changes SQL 2016 has brought (or will bring soon in some cases), which eliminate problems ranging from small code changes to entire systems:

Native string split function
Query Store
Always Encrypted
Polybase
Power BI embedded in SSRS

I've been involved in projects which have built, maintained, and ultimately deprecated various solutions which are all solved far more elegantly by something which is now part of the 'boxed' product. It's so easy to delegate the responsibility of knowing this to architects/consultants, but the art of the possible is something each engineer needs to own for herself.

The key to brilliance when being broad isn't knowing everything that is happening (impossible) or the fine details of everything you learn (also impossible), but to have a model of the system you're trying to understand, and a method for keeping up to date or digging deeper.

Very briefly, this might mean you split your ecosystem into a few areas (perhaps by product, by problem solved, by team). For each area you would think about how you'd keep abreast of significant happenings in those areas (that might be colleagues, conferences, blogs, etc.). Finally you'd assign yourself some kind of depth gauge for each area - if you're spending no time keeping up to date with .Net and 100% of your data access happens in .Net, you'd want to be very careful before relying on any old assumptions.

Ownership of problems & their solutions end to end

First and foremost we're problem solvers. The best problem solvers I've worked with have all shared the same fundamental trait - they own the problem. I'd hesitate to call the following a framework, but I've seen very similar steps executed by some of the most respected problem solvers I've seen in action:

Understand the problem
- Why are we solving this problem
- What will we gain
- How will we know if we've solved the problem
Understand how the problem might be solved
- Explore a few different options, even if unfeasible now (what might another solution look like?)
Understand how this particular implementation will work
- Any weaknesses, positive-side-effects
- What are the unknowns
Identify all the stakeholders
- Who needs to know before we start/as we progress/when we're done
Own the problem until it's solved/no longer a priority

Implicit in the above (but explicit in the title of this section) is that you're not just taking ownership of the data/data-tier. This can really manifest when exploring other options - you might not be empowered to make process or application changes, but if fixing bad data can be better solved by modifying the application or business processes that input the data you owe it to the data to at least outline that option. I'll talk more about the data (and the system) as a customer later.

Do not tolerate badness

Unpacking that a bit, there are two ways in which this tendency will typically show up when working with an exceptional engineer:

She ships high quality solutions (quality being not-bad)
When she discovers badness she has an almost pathological desire to fix it
Any system she works on typically benefits from her attention in either uncovering or fixing badness

Obviously terms like badness and quality are subjective, and even when defined there is a scale. It gets even worse when you layer on the fact that your primary goal will typically be to ship, and so stopping to fix every bit of badness you find would leave you paralysed and unable to deliver anything. Cutting this Gordian knot with the appropriate degree of pragmatism is a lifelong challenge for any engineer that values shipping & disdains badness.

Practically speaking, you should understand both personally and as a team what constitutes badness, and then call it out whenever you see it. Given your wide range of customers (see the next point) there will invariably be badness in your systems - how you choose to deal with it and when will shape the future of your system.

Consider an example of a backup failure. Standard response might be to re-run the backup. A more curious engineer might dig and perform some root-cause analysis. A great engineer might automate the response. An exceptional engineer might keep digging until she discovers the underlying badness that leads to the backup failure and do what she can to fix it, or expose the issue to the relevant team (whose problem would the SAN backup target running out of space be?).

Treat the system as a customer

While everyone may care about the data, the scalability, and the performance of the [database] system as it pertains to their application, as a data engineer you are ultimately accountable for all of those concerns, for every application. As you're making changes you should not only be thinking about the end-customer and your internal customers (colleagues), but also the system that you're modifying & maintaining.

Building a healthy level of scepticism will set you in good stead for ensuring you keep your data systems correct (free from corruption), available, and performant. As the feature comes in which is going to add one call per second to some component, you'll be thinking about what happens at one and two orders of magnitude above that (and if you don't, who will?). When you get access to your SQL Azure instance that supports 1000 TPS, you'll be the one verifying that and finding out what happens when you exceed that number - perhaps even coming up with a list of options for what you could do in that event.

There are many ways a given system can start to go south, most of which won't manifest as an application issue until it is too late. This means you and your team have to monitor and be responsible for understanding the state of the system independent of the applications they support. Your goal should always be to have the system alert you (or self-heal) so the problem is solved before it bubbles up to the application.

Unlike most other customers the system doesn't have a way of making these problems known (colleagues and customers are typically adept at voicing their concerns), so in treating the system as a customer you have to ensure that you've put the appropriate structures in place to get useful feedback and insight into the current/historic state of the system.

Summary & Additional Reading

Excellence doesn't happen by accident - it is something which you have to continually and deliberately work towards. I've outlined the template I currently use when having discussions about excellence in database engineering with others or for my own introspection. The list is really just a jumping-off point, and I hope to continue having interesting & challenging conversations with brilliant people about what excellence looks like to them.

Below are a handful of interesting articles/books which served as inspiration for parts of this post. If you have any questions or thoughts about what makes an excellent engineer (database or otherwise) drop me a line or leave a comment below.

Principles - particularly principle 138, which has made its way verbatim to my point 'Do not tolerate badness'

Site Reliability Engineering - Epitomises what I think exceptional operations looks like, and what anyone servicing a shared system (like maybe a database) should aspire to

Valve Staff Handbook (particularly pages 39-46) - Calls out T shaped individuals (breadth & depth) and hits a lot of the right notes on hiring for excellence

Chris Adkin, Paul White, Bob & Bob - When someone asks me for an example of an expert in a few different domains these are some of the blogs I send them

Impostor syndrome - Great article with a wonderful closing quote that only becomes increasingly relevant the more you learn. Excellence is a tall order, it's ok to feel a little overwhelmed by it all