Operational Excellence Means Automation

People use the term “operational excellence” in a lot of different ways. In its vaguest sense, it means continuous improvement as applied to operations. But you’re interested in what it means in the context of technology operations. And I’m here to tell you that it means automation.

Operational Excellence is one of the five pillars of the AWS Well-architected Framework. The AWS whitepaper lists six design principles for achieving operational excellence. I’ve paraphrased these principles for clarity. Here they are:

Define everything as code

This is easily the most obvious. Turn everything into code that can be automatically executed by a machine. This includes the building of infrastructure, application deployments, testing, recovery, and anything that requires or benefits from being defined in a runbook. If it’s a repeatable process, code it and let a machine do it.

Documentation as input and output

The delightful side-effect of defining everything as code is that code can serve as documentation. It becomes trivial to have a machine take code as input, execute it, and then generate some pretty documentation based on a template. The resulting documentation can then be used by another machine. All automatically, of course.

Changes should be as small and frequent as possible

Without getting into the rationale behind this, the point is that the only way to make small changes as frequently as possible is to use automation. Pushing a code change to a repo whence it’s automagically built, deployed, and documented is faster than doing any of that manually. Reversing a change automatically is faster, too.

Look for things to automate

If you’re not automating something, and you can, then do it. Of course, you should avoid automating a bad process. Fix the process and automate it. And if there’s nothing to automate right now, keep looking, because changes will inevitably bring opportunities for automation.

Inject failures

Break things to cause failures. If recovering from those failures requires manual intervention, automate the recovery steps.

Tell other people in the organization to automate

The idea is to share what you’ve learned with others. Of course, what you’ve learned is that automation is the key to achieving operational excellence. So just keep it simple and tell them to automate.

But isn’t operational excellence more than just automation?

What operational excellence actually looks like depends on the organization. But no matter how you slice it, you’re closer to operational excellence if you automate than you are if you don’t. So yes, there is more to it than just automation, just as there’s more to driving than going from point A to point B. But if operational excellence is the goal, you need the vehicle to get there, and the only vehicle that will do it is automation.

Using AWS Systems Manager to Upgrade WordPress

After years of manually upgrading my self-hosted WordPress installation, I decided it was finally time to apply some devops principles (namely automation) to this process.

This site runs on an EC2 instance on AWS, so I decided to use AWS Systems Manager (aka SSM). I started out by creating the following Command Document (which happens to be in YAML format because JSON is ugly):

---
schemaVersion: "2.2"
description: "Download and install WordPress"
mainSteps:
- action: "aws:runShellScript"
  name: "example"
  inputs:
    runCommand:
    - "wget https://wordpress.org/latest.zip"
    - "mv latest.zip /var/www/html"
    - "cd /var/www/html"
    - "service httpd stop"
    - "unzip -o latest.zip"
    - "service httpd start"
    - "rm -f latest.zip"

The Command Document executes the bash commands in the runCommand section. It downloads the latest version of WordPress, stops Apache, unzips the files, restarts Apache, and then cleans up.

SSM uses an agent to carry out the bash commands. My instance runs Amazon Linux which comes with the agent preinstalled, so I didn’t need to install it.

Systems Manager can execute the Command Document at regular intervals to keep up with the typical WordPress release schedule of every 1-2 months. I can also trigger it manually if there’s a security or bugfix release I need.

To avoid catastrophe, I have the Amazon Data Lifecycle Manager for EBS Snapshots take daily snapshots of the instance, just in case something goes terribly wrong with an upgrade.

Visual Studio Code as a PowerShell Integrated Scripting Environment

I know what you’re thinking. “Why use Visual Studio Code instead of the PowerShell ISE?” Well, if you’re using Mac OS or Linux, you don’t have the option to use the PowerShell ISE natively. And that’s a problem if you want to take advantage of the cross-platform capabilities of PowerShell Core. In this article, I’ll show you how to use Visual Studio Code (free!) to perform the key functions of the PowerShell ISE, namely:

  • Simultaneously view code and execute it in the PoSh terminal
  • Execute code on a selection or line-by-line basis (F8)
  • Syntax highlighting (for people who are easily bored like me)

Installing Visual Studio Code

The best way to install most things is with a package manager. This would be something like apt-get or yum for Linux distros, homebrew for Mac OS, and Chocolatey for Windows. Or you could go old school and download it here.

Installing the PowerShell Extension

Go to the Extensions button (looks like a busted up square) or View > Extensions.

If the PowerShell extension doesn’t show up under the bombastic “RECOMMENDED” heading, just search for it in the “Search Extensions” field. Then install it.

Integrating the PowerShell Terminal

Open up a PowerShell script of your choice. Then in the menu, go to View > Integrated Terminal. You should see the following.

If you don’t see the PS prompt Make sure you select “TERMINAL” and “PowerShell Integrated” from the drop-down menu.

Running Only Selected Code

In the PowerShell ISE, you can select a block of text and hit F8, and the ISE runs only that code. Or you can position your cursor at the end of a line and hit F8, and the ISE runs only the code on that line. Next we’ll enable the exact same behavior in Visual Studio Code.

Go To File > Preferences > Keyboard Shortcut

In the text entry field at the top, type “runsel”. You should see two items:

  • “Run Selection” with a keybinding to F8
  • “Run Selected Text In Active Terminal” with no keybinding

This is not what we want because it will not run the selected text in the PowerShell terminal. It will run the selection in the “OUTPUT” section, but not in the terminal. Obviously, that’s not the normal behavior of the PowerShell ISE. Let’s fix it.

Right-click the “Run Selection” item and select “Remove keybinding”

Right-click the “Run Selected Text in Active Terminal” item and select “Add Keybinding”

Depress the F8 key (or whatever you want to use) then hit Enter.

Testing It Out

Go back to your code and select a block of code. Hit F8 and watch the magic!

That’s what I’m talking about!

But… as of this writing, there’s an issue with this that’s being tracked on the vscode-powershell GitHub repo, and it’s this: multi-line input in the integrated console doesn’t work. That means you can’t select a function block, hit F8, and have it work. It will throw ugly errors in your face.

Ready to learn more PowerShell? Get unlimited access to Pluralsight’s entire course library!

Building Windows Server with Puppet and Chocolatey

Forget using scripts and group policies to configure a new Windows Server machine. Using Chocolatey and Puppet, you can do it faster & easier than ever (and it’s more fun too). This is especially true if you’re using a Server Core installation and don’t have a GUI to help you along. Oh, and if you don’t know Puppet, you really should watch my course Puppet Fundamentals for System Administrators on Pluralsight 🙂

Assign IP address using PowerShell:

$ New-NetIPAddress –InterfaceAlias "Ethernet" –IPAddress "192.168.51.29" –PrefixLength 24 -DefaultGateway 192.168.51.8

$ Set-DnsClientServerAddress -InterfaceAlias “Ethernet” -ServerAddresses 192.168.50.20, 192.168.50.21

Install Chocolatey:

$ set-executionpolicy unrestricted
$ iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))

Restart PowerShell

Install VMware tools

$ choco install vmware-tools

The server will automatically restart.

Rename server

$ rename-computer -newname newservername

Reboot

restart-computer

Join to domain
add-computer -domain benpiper.com

Reboot again
restart-computer

Install Puppet
choco install puppet

Configure Puppet
Configure c:\programdata\puppetlabs\puppet\etc\puppet.conf

Generate puppet certificate
puppet_interactive

Sign puppet certificate on puppet master
puppet cert sign newservername

Apply appropriate profiles to server. Remember to restart the Puppet master if you change your Hiera configuration.

Run Puppet agent
puppet_interactive

Verify
puppet resource dism
puppet resource package