WindowsNinja

The musings and mishaps of a Windows sys admin

latest news

Leave your pagers at the office – It’s the official Baltimore/DC Sys Admin Day Party!

Following in the footsteps of our fellow LOPSA chapters up in Boston (a big thanks to Matt Simmons for leading the way) and Columbus, Ohio, I’m proud to announce a joint party to celebrate Sys Admin Day put on by the Crabby Admins (Baltimore’s LOPSA chapter) and LOPSA DC in addition to the Baltimore SpiceCorps chapter!
 
I’ve tried to pick a location that is conveniently located between Baltimore and DC though I realize this isn’t actually possible. In any case, all the pertinent party details can be found below.
 
Date: Friday, July 26th
Time: We’ll be getting started at 6pm but there’s no telling when we’ll be finished
Location: Looney’s Pub in Maple Lawn, MD
 
If you haven’t been to Looney’s before then you’ve been missing out. This is one of the best spots in the DMV region…what are you waiting for?
 
Just in case you need some additional persuading we’ve got some pretty sweet gifts lined up for attendees as well!

  • Appetizers are being provided by our friends over at Spiceworks
  • O’Reilly has agreed to provide a free eBook of your choice to all attendees

 
Additionally, I’m working on lining up a drink sponsor as well. Nothing is confirmed as of yet but with a bit of luck and some much appreciated generosity I’ll be able to get this lined up as well. Check back here as Sys Admin Day approaches for updates.

 

Tickets are free – the only thing required is your attendance! Get your tickets HERE.
 
Looking forward to seeing you there!

The Cable Stash (aka Memories of a Lost Age)

I’m currently in the process of moving and it has been a lot of work. Tonight I’ve spent my evening cleaning out the storage closet in my office and more specifically, a large trunk I have within that closet.
 
This trunk contains cables of every variety that I’ve collected over no less than ten years. Going through these has prov ided me with a rather unpleasant self-realization: I’m a cable hoarder.
 
My wife and I work very hard to eliminate all clutter from our lives and we generally do a pretty good job of it. However, this trunk is a glaring exception for me. Going through this treasure trove of cables I’ve found every type of cable a techie like me could ever reasonably need. Amongst the most memorable:
 

  • Every type of USB cable imaginable (and multiples of almost all of them) 
  • Parallel cables – you know, just in case I really need to hook my computer up to a 1990s printer
  • Enough IDE and SATA cables to provide storage to a small country
  • Power cables in every different variety (it’s amazing how many different styles of plugs they make)
  • A/V cables for almost all of the last 3 generations of video game consoles
  • More coaxial cable than they’d need to distribute cable to every room of the White House

While this has certainly been a trip down tech memory lane, I’ve resigned myself to admitting that I’ll likely never need the majority of these cables ever again. With that said, I’m proud to announce that I trashed two-thirds of these cables and only kept the bare necessities that I will likely use again some day (HDMI cables, a few micro-USB cables, etc.).
 
So let’s hear it fellow techies, what’s in your cable stash?

IT – Cost Center or Insurance Center?

A reddit poster, /u/asdlkf, offered a very interesting perspective on the current state of IT yesterday. He (or she) hypothesized that the days of IT  being purely a cost center are in the past. These days companies need to view their IT staff and infrastructure as an insurance center, a piece of the company which offers varying levels of assurance and protection against outages depending on the amount of money they are willing to invest.
 
This really got me thinking about things and how absolutely correct this person was about how companies should approach IT. From the perspective of a company executive it is very important that they not simply look at IT a necessary evil costing the company precious dollars. Instead, they need to view us as an asset which provides assurances on the company’s most important assets. When determining budgets for their IT expenditures, it is not enough to simply calculate a few salaries with minimal support in terms of hardware and software licensing, expecting them to make miracles. These executives really need to consider how vital their core services are.
 
Consider this: How much would it cost your company if there was an 8-hour outage for your company e-commerce website? Once you’ve determined how much something like this can potentially cost your company, you next need to determine how much you’re willing to invest to prevent this from occurring.
 
Let’s use this as a hypothetical example:

  • E-Commerce Site: Outage costs $100,000/hour
  • IT Insurance Solutions:
    • Scenario A: Cost to implement solution with no redundancy – $50,000/year
      • Chance of 1-hour downtime per month – 70%
      • Chance of 3-hours downtime per month – 20%
    • Scenario B: Cost to implement solution with n+1 redundancy (all at the same site) – $250,000/year
      • Chance of 1-hour downtime per month – 10%
      • Chance of 3-hours downtime per month – 3%
    • Scenario C: Cost to implementation solution with n+1 redundancy onsite and an additional n+1 solution in a geographically diverse area – $500,00/year
      • Chance of 1-hour downtime per month – 1%
      • Chance of 3-hours downtime per month – .1%

The math above is extremely simplified but this will allow us to show how IT solutions can be presented in terms of an insurance center. A company’s finance/risk team can then perform some calculations to determine how much money to invest in IT to  determine the proper amount of risk that should be taken.
 
For example, when looking at the potential for a 3-hour outage you could use the math below:

  • Scenario A - Cost to reduce likelihood of 3-hour outage to 20% is $0 (as $50,000 is the minimum to provide support for the solution and is therefore not additional cost)
    • $100,000/hour * 3 hours = $300,000 * 20% likelihood = $60,000/month average risk
    • Annual Expected Cost of Downtime – $60,000/month * 12 months = $720,000
  • Scenario B - For a cost of $200,000 ($250k minus the $50k mandatory cost) you can reduce the likelihood of a 3-hour outage per month from 20% to 3%.
    • $300,000 * 3% likelihood = $9,000/month average risk
    • Annual Expected Cost of Downtime - $9,000/month * 12 months = $108,ooo + $200,000 (upfront cost) = $308,000
  • Scenario C - For a cost of $450,000 you can reduce the likelihood of a outage from 20% to .1%
    • $300,000 * .1% likelihood = $300/month average risk
    • Annual Expected Cost of Downtime - $300/month * 12 months = $3,600 + $450,000 = $453,600

 
Using the math above you can perhaps conclude that the best combination of cost and risk is Scenario B (which would expect to cost $308,000/year vs. $720,000 for Scenario A and $453,600 for Scenario C). Obviously, this is highly simplified and does not include all the necessary calculations for a real scenario (such as what is the risk of 10 hours downtime per month?). However, this gives you a good starting point to begin working with your management team on the importance of viewing IT not simply as a cost center but as an insurance center which offers varying levels of coverage based on their investment.
 
 
It is incredibly important for us as engineers to look at things not only from a technical perspective but also a business perspective. When communicating with the business about these different risks, we need to put it in terms they understand. If we fail to do this and we’re not given the support we desire then the failure is on us and not them. However, if you do provide them with this information then the burden is then shifted upon the business as it is their own assessment of the potential risks that made this decision.

 
This is one of the most challenging things to do as a technical person, to step outside our own world and work within the realm of another. However, this is the difference between a good engineer and a great one. A good one can build you a system that will give you 5 9′s reliance. However, a great engineer can design a system with 5 9′s reliance and then sell this system to the decision makers.

 

Always strive to be great.

I Accidentally Increased the Disk Size of a Hyper-V 2008 VHD with Snapshots Present…now what?

I just managed to get myself out of a sticky situation caused by some hair-brained execution on my behalf. At $work we’re at the tail end of a migration from Exchange 2010 On-Premise to Office 365. As part of this migration, Microsoft just upgraded us from Exchange 2010 to Exchange 2013 on their systems which has caused some compatibility issues for us as we’re currently operating in a hybrid configuration where we still need to be able to manage a number of things from our On-Premise Exchange Management Console.
 
The only problem is our On-Premise configuration is on an outdated version of Exchange 2010 and only Exchange 2010 SP3 is fully compatible with Exchange 2013 on Office 365. What this means is that we’re not able to connect to the Exchange 2013 tools from our Exchange Management Console until we upgrade things to Exchange 2010 SP3.
 
Still with me? Good.
 
A screenshot of the error we were receiving when trying to open the Office 365 EMC within Office 2010 On-Premise is below.
 
EMC Error
 
So, off I went upgrading our On-Premise Exchange environment to Exchange 2010 SP3. The first step in doing so is upgrading your CAS (Client Acccess Server) servers to SP3 before the mailbox. Easy enough…only the first one I started working did not have enough space remaining on the hard drive. Fortunately this system was a VM so I just needed to expand the VHD (and then expand the partition within Windows Server 2008). Easy enough. Except it wasn’t so easy.
 
Enter scene: Hyper-V 2008 Manager
 
It is a big No No to expand a VHD of a system that has an existing snapshot and I knew this. To ensure there were no existing snapshots I opened Hyper-V Manager and right-clicked on the desired VM in Hyper-V Manager and selected “Snapshot” just to see if any snapshots existed. However, it didn’t do anything after doing this so I looked in the section of Hyper-V Manager where the snapshots would be present and could see there were none.
 
With this knowledge in mind I went ahead and proceeded to expand the size of the CAS server VHD from 40gb to 60gb. When this finished, I noticed that a snapshot DID appear for the server now. Earlier when I clicked snapshot, even though I was presented with no options and wasn’t asked to confirm anything, it proceeded to create a snapshot (seriously Windows? what was the thought process here? I guess that is what I get since I’m used to VMware ESXi where you have to name your Snapshot and confirm that you want actually take a snapshot before it does anything).
 
Thinking that this wasn’t a huge deal I went ahead and deleted the snapshot since I didn’t intend to create one anyways. Problem resolved.
 
Only it wasn’t…the expanded disk was created and it was done with the assumption that there was a snapshot chain in place. Starting the VM after this presented the error below…uh oh!
 
VHD Chain Corrupted
 
After some initial panic I reminded myself to stay calm (important!) and devise a road to recovery. I would spend an hour trying to figure out a way to recover from this mistake and after that I would just deploy a new CAS server to replace it. This would be a time sink but one I could recover from.
 
Enter scene – VHDtool.exe (source: http://archive.msdn.microsoft.com/vhdtool)
 
This tool saved the day for me. I wasn’t 100% sure how to use it initially but after playing around with things I was able to get it to work and was able to get this problem fixed without having to resort to deploying a new CAS server!
 
The details below will outline what you need to do to get this tool to work if it is ever needed again the future (hopefully not).
 

  • Initially I was controlling Hyper-V from client machine and this did not work as the VHD was not located on this system. So the first step is to remote onto the actual Hyper-V server and then download VHDtool.

 

  • Once I downloaded the tool, I place the file in c:\Windows\System32 so that I could access it from the command line without having to input the directory

 

  • The next step was to figure out the name of the base VHD and the name of the snapshot it was expecting to be part of the chain, in addition to where they’re located (the easiest way to do this was to select the virtual system in Hyper-V manager and then select Edit Disk and the Browse to see where the files are).
     
    VHD File Name – basefilename.vhd
     
    AVHD File Name – you’ll find this file in the error screenshot above as well as from the file name in the folder where the VHD is located

 

  • Once I had figured this out I opened an administrative command prompt (right-click Command Prompt and select “Run as Administrator”). From there I navigated to the location of the VHD and AVHD files> cd c:\ClusterStorage\Hyper-V\CAS2

 

  • Next I ran the below command to repair the VHD> vhdtool.exe /repair ex2010-cas-fixed.vhd  ex2010-cas-fixed_37E0C9A9-11CE-46F6-8BAA-E0326EB3EF73.avhd

 
That’s it! Once I ran the command above everything started working again! It was a bit challenging to track this solution down but now that I know what to do it will be simple in the future (though hopefully it never happens again).

The Art of Problem Solving – Documentation

Product documentation should be your starting point when working with any technology. Documentation comes in many different forms and some of it is vastly better than others, but it is a good starting point any time you are working toward solving a problem. While the usefulness of documentation may vary, the biggest hurdle is determining how to find the documentation you’re looking for.

Windows Documentation

When searching for documentation for literature provided by Microsoft, there are a few places you should look:

  • Microsoft Technet is the primary resource for documentation on any software solution provided by Microsoft. To find the documentation you need, Google whatever product you’re working with and append “+technet” to the end of it. Additionally, you can perform search queries directly on their website.
  • PowerShell - Get-Help is your go to resource for documentation any time you’re working with PowerShell. To retrieve a list of all available help topics, simply type “Get-Help *” and it will provide you with a long list of commands that help is available for.
  • Command Prompt – append “/?” to the end of the any Windows command you’re trying to use and it will provide you with some basic instructions in addition to a few examples

*nix Documentation

Documentation for *nix based systems are available in much the same way as it is available for Windows based systems.

  • Distribution documentation – Each distribution has their own website dedicated to documenting their systems. Some of the more common ones are available below:
  • Man pages – To read the documentation to any *nix commands you simply enter “man <command name>”at the shell prompt and you’ll be presented with all of information you’re likely to need for that command.

 

A good grasp of using documentation is the foundation for achieving greatness in the world of systems administration. Master it early and use it often!