Let's talk about Nano baby!

Just watched a demo of a Windows Server 2016 Nano vm deployment using PowerShell by one of our very own DevOps Dudes (Si - you da man)...and it rocked! I absolutely love the idea of Nano and to see it up an running within a few minutes with a few lines of Posh code...well, let's just say the brain was on fire!

If you haven't looked into Nano yet, you might want to take a look at Channel 9's dedicated channel: https://channel9.msdn.com/Series/Nano-Server-Team

This is the future dudes and dudettes!

CredSSP...say what?

I'm working on a DSC configuration for SQL Server Reporting Services and it is quite a simple configuration with very few required resources. However, one of the resources (xSQLServerRSConfig) does something rather odd in it's Set-TargetResource function - it uses Invoke-Command to loop back to the same server and specifies the -Authentication parameter as CredSSP.

Now up until the other day I had no idea what CredSSP was and the exception that was raised by the LCM was rather new to me:

After spending ages trying to learn what the CredSSP protocol is and why someone would use it (see https://4sysops.com/archives/using-credssp-for-second-hop-powershell-remoting/ and https://technet.microsoft.com/en-us/library/hh849872.aspx ) I decided to check for a DSC resource to enable it...and there is one, xCredSSP. Happy days!

Then came the joy and agony of seeing DSC in action. Joy because the resource is so simple to configure:

xCredSSP Client
{
 Ensure               = 'Present';
 Role                 = 'Client';
}

Or

xCredSSP Server
{
 Ensure               = 'Present';
 Role                 = 'Server';
}

Seems simple enough, and it just works (the resource that is) but the resource that needed it still failed with the same error. I tried setting my node as the client and then as the server and then as both (by specifying both configurations above) and still nada, nothing, zip - the exception was the same.

Then finally (after getting on my knees) it dawned on me that I was right to specify the node as client and server was the right thing to do because of the loop back nature of the Invoke-Command (notice the . after -ComputerName in the example above) but what was missing was the delegation rights that the xCredSSP resource needed to apply. So I added the following:

xCredSSP Client
{
 Ensure               = 'Present';
 Role                 = 'Client';
 DelegateComputers    = 'myvm.mydomain.com';
}

Well, that did it. By telling the security provider that implements CredSSP what servers were allowed to be delegated to, it just worked. It is worth noting that the DelegateComputers property supports an array of computer names or even the *.mydomain.com wildcard.

Getting to grips with Git

If you are new to version control or have grown up using something other than Git (like TFS) then this video Build Conference video is going to help a lot!

Be warned, there are some very important conceptual differences between Git and TFS, so watch it dude! (or dudette :-)

Who Ate All The Disk Space? (Yeah, It Was BizTalk)

This post is for newbie BizTalk users who have installed their environment using the Next, Next, Finish approach.

Today you have your shiny new development sandbox* with a nice chunk of disk space. Tomorrow you will have slightly less disk space. Next week a little less still. In a month, no disk space. This will be a bad place to be.

There are a number of things that eat up disk space in the world of BizTalk:

1. BizTalk uses a number of SQL Server databases at its heart. These databases are NOT backed up by default and you should NOT use database maintenance plans or third party backup software to back them up. You'll see why in a minute.
2. BizTalk normally reads messages, processes them, delivers them and then deletes them. It keeps the databases surprisingly small. However if your BizTalk solutions have issues the messages will be "Suspended" which means "saved to the database until the problem is fixed". At this point your database is now growing.
3. BizTalk normally reads messages, processes them, delivers them and then deletes them. This is great until someone asks you what happened to a message from, for example, the last fiscal month. At this stage you will turn on various tracing options, add logging to your process and add archiving ports to keep copies of messages. At this point you are likely to persist multiple copies of every message flowing through BizTalk which is going to get very big, very quickly.

Before you do any other work on your sandbox, get your house in order. I would heartily recommend performing these steps as a minimum:

1. If you use the BizTalk Administration Console to switch on tracking of messages for any BizTalk component, use it sparingly and then switch it off as soon as you have diagnosed your issue (later we will see how debugging works and how to avoid using tracking anyway). Never use the tracking options in your production environment.
2. Create a folder, for example, C:\FileDrop, to be the single location for all incoming and outgoing files on your sandbox. Then create a Windows Scheduled Task to keep that folder free of old, junk files.
3. Configure the BizTalk SQL Agent jobs that the BizTalk installer created and then disabled.

The rest of this post covers what to do with the BizTalk backups and why.

First of all BizTalk uses multiple databases. There are a number of reasons for this including the theoretical option of putting the databases on multiple servers. The downside is that a normal backup script will back up the databases sequentially, meaning you have no single point in time to recover two. The backups will be out of sync and potentially useless in the event of a disaster.

Out of the box, BizTalk provides backup jobs to work around this complexity.

1. Backup BizTalk Server (BizTalkMgmtDb)
2. DTA Purge and Archive (BizTalkDTADb)

The BizTalk backup jobs deal with this using a feature in SQL Server referred to as checkpoint marks. The first step performed by a BizTalk backup job is to create a single, simultaneous checkpoint mark in all the BizTalk databases. The next steps then back up each database up as at the checkpoint.

This is all pre-written for you in a bunch of stored procedures which are called from the two jobs I've mentioned above. However the job steps are not configured automatically. You need to edit the steps to put fill in a couple of parameters, namely the backup location and the retention period. Here's an example of how those steps may look for the Backup BizTalk Server (BizTalkMgmtDb) job.

• exec [dbo].[sp_BackupAllFull_Schedule] 'd', 'BTS' , 'E:\Backup'
• exec [dbo].[sp_MarkAll] 'BTS', 'E:\Backup'
• exec [dbo].[sp_DeleteBackupHistory] @DaysToKeep=2

Here we are backing up to the E:\Backup folder. We are backing up every day ('d') and we are creating a checkpoint mark called 'BTS'. We are also going to delete any logs over two days old.

These steps will back up all of your critical, frequently changing BizTalk databases. The stored procedures take more options but for a sandbox server these should do you well. For the "less critical" tracking databases here is an example of how to configure the DTA Purge and Archive (BizTalkDTADb) job.

• exec dtasp_BackupAndPurgeTrackingDatabase 0, 1, 7, 'E:\Backup'

These jobs will keep your databases backed up. There are now a couple of final tasks you will need to do.

1. Create a Windows Scheduled Task to delete old backup files
2. Periodically shrink the databases if required (unlikely and usually not essential)

Finally a little tip for you. How does BizTalk know what databases to actually back up when it runs these stored procedures? It has a list, stored in the database BizTalkMgmtDb. And, even better, it has an extra list for adding custom databases to the overall backup job. Now you can keep your own solution databases in sync with the BizTalk databases for perfect point of time recovery from disaster.

* If you have wedged BizTalk on a pre-loved, tried and trusted sandbox see my upcoming post, "Why Not To Wedge BizTalk Onto A Pre-loved, Tried And Trusted Sandbox"

* There are actually two jobs which deal with "priority" and "non-priority databases

Stuff about SCVMM Templates

Revisiting previous code projects can be an exercise in hair-pulling. For example, in my previous blog post I mentioned creating a VM deployment script. Due to some issues with binaries, the need to revisit the code has come up recently, to add an option for creating VMs with a GUI, rather than as Core. The code worked when it was just dealing with one choice, but adding a very simple -InstallGui switch turned out to be a little more complex than originally thought.

Constant errors stating the following were the main contention:

 New-SCVMTemplate : VMM is unable to process one or more of the provided cmdlet parameters. (Error ID: 1600)  

Needless to say, this is a bit non-specific! After a lot of frantic searching of blog posts and whatnot, and complete inability to find a relevant fix, I decided to scrape through the code and remove parameters one by one. Eventually, the Template parameter seemed to show up as a potential problem.

Hmm.

Inspecting the template I was intending to use, I noticed something - there was no OS Configuration section, therefore no ability to pass parameters such as Domain or DomainJoinCredential. Fixing this resolved the above error.

Now, on to simultaneous deployments!

What's up with my Config dude?

When you want to know the current configuration of a DSC node you use the very aptly named Get-DscConfiguration cmdlet like so:

PS C:\>Get-DscConfiguration -CimSession $target_node

This cmdlet will execute the Get-TargetResource function for every configured resource on the node and return details on its current state like so:

And life is good until you add a new resource to your configuration, push it to the target node and run the cmdlet again, only to be faced with an ugly exception like this:

I know what you're thinking, "say whaaaat!!". You just want to see what the current config is and because of one dodgy resource you can't see anything - that sucks. However, the solution is quite simple. The words that stand out are key is not a valid property in the corresponding DSC resource schema file. The preceding word (AvailabilityGroupNameDatabase in this case) is the property that the resource Get-TargetResource function is trying to set which is not defined in the schema.mof file for the resource. What you need to do is add the property to the schema.mof file or modify the Get-TargetResource function of the offending resource.

On learning new skills...

I'm your typical infrastructure guy - design the platform, run through the installers, use PowerShell to do post-setup and admin tasks, maybe script something to make a boring job easier - you know the drill. This DevOps thing is kinda cool to see working, but how does that fit in with the way I'm trying to do in my projects? As it happens, it fits in really, really well. We already use Agile methods in our project delivery team, and the Ops team have just adopted Scrum as well. The DevOps mindset isn't terribly difficult to adopt when you're already used to delivering small improvements often. What is different about all of this is, for me anyway, is the Dev part of DevOps.

I've had my PowerShell knowledge tested and expanded whilst working with this - first, by writing a deployment function to take what System Center Virtual Machine Manager does and making it fit into our deployment requirements and under source control; secondly, working on the SharePoint 2016 platform we've been tasked to deploy by using Desired State Configuration. I've had to learn about source control, about commits and why you attach them to work items, about injection, about the separation of infrastructure and application, the whole nine yards, and I'm not even halfway done, if you follow some of the guidance out there. 

One of the things I've learned during this entire process is that red in your PowerShell console isn't a Bad Thing. Not even close - it's usually helpful when dealing with a complex beast like SharePoint. Granted, some of the errors during testing haven't been PowerShell-based, and some have been really, really odd. 

For example, the Event Viewer on the primary application server logs Event ID 3351 in the Application Log (SQL Error 18456, State 5 (Invalid user ID) in the SQL logs), stating that the SP Farm account is a bad login. However, the account is present as a security principal in the SQL instance AND the SP_Config database. What? 

Turns out the issue was caused by the script execution speed beating Active Directory replication and adding in a tombstoned SID to the database. Slowing down the reset process during testing was all that we needed, but it caused a lot of head-scratching! 

But what we do is hard, right? As my esteemed colleague said to me, if you run it and it goes perfectly, first time - what did you learn? The fact that we can review and change the code, chipping away at the problem one red line at a time, that we can repeat it again and again and again until we get it right - I've found that it's really important. We learn by doing. 

I've been involved with this DevOps methodology for a month now, and I'm enjoying every minute of it. 

Do you know what a Paradigm Shift is?

Well, it'll take you 5 seconds to google it but here you go:
Paradigm Shift: a fundamental change in approach or underlying assumptions.

The first time I encountered that phrase was with the release of Visual Studio Team Edition for Database Professionals. This excellent VS add-in was the very first tool that enabled SQL developers and DBAs to put their database schemas under source control using an integrated development environment and to validate the schema before deploying it to a server! This was something that application developers had been working with for ages and provided the opportunity for the database to join all stages of the software development life-cycle. The problem however, was that SQL developers and DBAs were not accustomed to using source control, to 'compiling' their code or to 'kicking off a build'. What was needed was a radical some significant upskilling and a complete change of mindset - they needed to start thinking like application developers.

DevOps is to infrastructure and operations guys the same kind of shift in thinking and skills. The things you need to learn are:

• obviously PowerShell - including how to write advanced functions, modules and package management concepts
• Desired State Configuration - including class-based custom resources, composite resources and composite configurations
• Source Control - which should really be Git rather than TFS, if you want to be in with the crowd
• Unit Testing with Pester
• Continuous Integration
• Release Management

...and that is not exhaustive. However, if you are thinking "no way Josẻ!" then just hold your horses dude, it ain't that bad. Just start with PowerShell and slowly build up your knowledge; within a few months you'll be cooking with gas!

As for the mind-shift, well that is really down to you and how adaptable you are to change. This is where I recon most will struggle. Building a pipeline for automation is easy to understand (if not to implement) and difficult to argue against (as anything else must involve repetitive, error-prone manual effort) but continual, frequent releases and relying on open source DSC resources to configure infrastructure! If  all the other stuff hasn't pushed your neurons to the limit then the obvious risk of these two approached may just push you over. But this would be an error in judgement because the risks involved in the alternative are far greater.

Think about it this way, which is riskier: to deploy a large change including months of effort using a mostly manual process that is difficult to repeat exactly or to deploy a small change using an automated, 100% repeatable process? No rocket scientist required.

And as for relying on community based, open source PowerShell resource modules to install and configure infrastructure...ask yourself which is better: to manually install software using wizards that are mostly inflexible and time consuming or to use pre-written PowerShell scripts that are readable, easy to edit and must faster to use?

So far we have used about 10 DSC resources modules to deploy a clustered SQL Server and SharePoint and so far I have found 4 syntax errors in 3 different modules - but the thing is, they are easy to find and to fix and I have the opportunity to contribute to their improvement using GitHub. That is a powerful mechanism for improvement given the hundreds of thousands of PowerShell users there are in the world.