ITMU update goes smoothly

The last week has been pretty uneventful. Yesterday, I installed the ITMU update on my test server. It updates the packages, collections, and advertisements as expected. It also adds in the 64 bit ones as well which will be nice if we ever let our users run the 64bit OS I suppose. The only disappointing part is that nothing in the install is configurable. We don’t use the default collections or advertisements, so we’ll have to update these on our own I suppose. Not a big deal. So I send out the change control and let everyone know that the clients will receive an updated Windows Update Agent (WUA).

Today we went ahead with the ITMU update on the production environment. Once again it was uneventful. The only minor detail was that we changed advertisement related to the scan package (Microsoft Update Tools) to use a test collection. One can never be too careful! Note, this package has the dependency to run the WUA package first. The WUA package should only run on computers it has not already run on, so in theory it will only run one time. Hope we don’t have too many failures. In any case it was no big deal. We updated the ITMU, and then updated the distribution points. We did note was that the old admin was not using every site server as a DP for this package. Since we didn’t see a reason not to have the package on every DP we fixed that issue. One final note is that I couldn’t find evidence of the package running on my test system. But mine already had the updated WUA so maybe it did. I have the other admin check his test box tomorrow and then we’ll change the advertisement back to use our normal collection.

Day 9: Heartbeat Discovery

So I’m ready to install ITMU on my test setup and I get to the point where it asks for a sync client. I decide to use the site server (against Microsoft advice, but it’s the way our production environment is currently configured). My problem is that the ITMU setup says that my site server is not a client. I double check the All Systems collection and I see it. But upon closer inspection the Site Code column is empty and the Client column says No. In fact is also says this for my actual xp client machine that is in the test domain. So I run ccmsetup.exe from \\siteserver\smsclient\i386 . Client is installed. Start>Control panel>Systems Management. On the advanced tab, the currently assigned to site code field is empty. I click discover. Viola! It discovers my test site. Switch back to the SMS console>Collections>All Systems. No change. How disappointing. F5. No Change. More disappointed. Right click>All tasks>Update Collection Membership. No Change. This is starting to bother me. Right click on the client>Trigger Discovery Data (you need the right click tools installed for this option to be available). Get the Right click tools from Stuart Watret at http://offshore-it.co.uk/ . Still No Change. Back to the client>Systems Management>Actions tab> Hardware Inventory Cycle> Initiate Action. Come back an hour later. No Change. I go back and forth before I put my time tested Googling skills to use. Google: collections client no sms installed. 4^th link down has the answer. It’s the heartbeat discovery. Not really sure how it differs from the AD System Discovery but it does. SMS console>Site Settings>Discovery Methods>Heartbeat Discovery>Right Click>Properties. It’s enabled, but by default set to once every 7 days. Since this is my test site, switch it to once every hour. Back on the client side> actions tab> Discovery Data Collection Cycle> Initiate Action. Back to the Site Server. Collections>All Systems>All tasks>Update Collection Membership. Ahhhh. Sweet SMS goodness. Instantly the two systems show up as assigned, client = yes, client type = advanced, etc. It all looks good. Back to ITMU!

I really need to get the SMS administrators book and figure out how all this should work.

Day 8: Critical Error 4909

So today my test site server was showing critical error 4909 with description:

SMS Systems Management Server could not locate the "System Management" container in Active Directory. Nor could it create a default container. This will prevent Site Component Manager from updating or adding any objects to Active Directory.

Possible cause: This site’s SMS Service account or the site server’s machine account might not have the correct rights to update active directory.

Solution: Either give the Service Account rights to update the domain's System Container, or manually create the "System Management" container in this domain's Active Directory system container, and give the Service Account full rights to that container (and all children objects.)

Since I’m knew at this stuff I had to figure out what it meant to give rights to the system account in AD.

Well after a bit of googling (googled: create the System Management container, 5^th link down), I came across this Microsoft KB article.

http://support.microsoft.com/kb/830022/en-us?spid=2890&sid=global

Step by step instructions. Basically in AD, you have to give the computer that SMS installed on, rights to the “System” folder aka container. Make sure you give it permissions to all child objects as well. If you’ve done it right, there is a new container object under System called…..System Management

The end of the instructions instruct you to Restart the SMS Site Component Manager service to start updating Active Directory. Of course Microsoft fails to mention how to do this, and the only people who need to know are the ones who don’t have SMS working i.e. meaning people who just installed it and may not know how to check a SMS service. Guess I can be nice and tell you. Back in the SMS console expand the site database>Tools>SMS Service Manager. You’ll see an empty pane. Very intuitive isn’t it? Anyways right click in the empty pane>All Tasks> Start SMS Service Manager. Expand your site>Servers> Machine name of your site> Pick the SMS_SITE_COMPONENT_MANAGER. Now things get really silly. You can’t tell if it’s running or not unless you query it first. Pick the component from the rights side and hit the query (!) button. Now that you can see it running, you can stop it. You’ll have to query it again to see that it actually did stop. Then you can hit Start, query it again to make sure it started. Now switch back to AD and check your new System Management container to see if SMS has put anything in there.

Day Seven: Test Environment is working

Well I finally got the test environment up and running. Had a little trouble because I forgot to change the SID on one of my VM machines. I took a shortcut and copied the VM machine folder instead of reinstalling windows 3 times. This of course caused trust issues with the domain since all the workstations had the same SID. Symptoms of this phenomenon include not being able to login to the domain on the workstation, and the domain administrators group not being included in the Local Administrators group of the workstation. The quickest way to resolve this is to move the machine back into a workgroup, Reboot, Run NewSID (http://www.microsoft.com/technet/sysinternals/Utilities/NewSid.mspx), Rejoin the domain, Reboot (again), and try to login using a domain account.

Sidenote: NewSID is an excellent tool developed by Mark Russinovich and Bryce Cogswell. It’s an interesting fact that Microsoft does not actually support the use of NewSID despite is availability from the Microsoft website. This is due to the fact that NewSID was developed by Mark and Bryce under their own company, Winternals. Microsoft bought them up and allowed them to rerelease their tools on Microsoft Technet with the disclaimer that it is not actually supported by Microsoft. Personally, I’ve used NewSID quite a bit and have never had an issue. However, if you wish to do things the right way, then Microsoft recommends running Sysprep. This tool will actually do lot more then generate a new random SID, so I’d advise just using NewSID.

Back to SMS and my test environment. Tomorrow I plan to apply SP2 to SMS and install the first revision of ITMU to accurately reflect the production environment. A quick patch update test should show everything working the same as it does in production. Then apply ITMU ver. 3 and another update test. If all goes well, I’ll be updating ITMU on the production server real soon.

Day Five and Six: ITMU and the Test Environment.

Hey can you look into updating to ITMU v. 3? I think we need it by March. I actually got this email from admin no. 3 a couple of days before we did the patch update. It was the first time I had heard of the Inventory Tool for Microsoft Updates aka ITMU. Researching what it was and why it need to be updated by March actually really helped out with the patch updates we did, since I had some idea of how Microsoft intended the monthly updates to be deployed. Guess what? It really does need to be in place by March. Microsoft will stop updating the old one and ITMU will break if we don’t update. Lucky me, I get a chance to break our patch system within the next 2 months. Actually, I don’t think it will be a big issue. We already have ITMU installed, so we meet all the current prerequisites. Version 3 has the same requirements, so we’ll just need to walk through the install. However, being the good little admin that I am, I pushed that we needed to walk through it on the test environment first. No problem, since it’s all on Virtual Machines. Well it wouldn’t have been a problem, until I found out that the old admin wiped all his boxes the day he left. His SMS test environment was completely lost. No DC, no parent site server, no child site server, and no clients.

Needless to say, days five and six have been an excruciatingly long effort to recreate the SMS test environment. Everything from installing Win2k3, Active Directory, Creating user accounts, Installing SQL server, IIS, BITS, SMS, SMS SP2, etc. You have to extend the AD schema? Give the SMS computer permissions to AD? Who came up with this insane procedure? I’ve installed SMS before, but I always hate doing it. I keep promising myself, this will be the last time. I’ll make nice backups of my VM’s and find some place to store them. Yeah right, in 9 months I’ll have a need to start over. Something will break and it’ll be easier to do a clean install than fix it. I didn’t quite finish today, so the whole thing will be waiting for me tomorrow. I hope I don’t hit any walls.

Day Four con’t: The 2nd meeting

The second meeting we had yesterday (why does every company have so many meetings?), involved scheduling the patches for the rest of the company. At our company we actually send out 3 packages at once. The first package is the patch notice. It informs our customers (IT world translation: end-user) that their machine is part of the SMS patch program and is targeted to receive the latest patches. It also gives instructions for calling helpdesk (I’m sure helpdesk is thrilled that we give that extension out every month to every customer). The notice is available starting at 8:00am and the advertisement runs for 8 hours (5:00pm). If the customer clicks okay, or 8 hours passes and the notice closes itself, the program package ran successfully. At 11:00pm, the Microsoft Patch Update program runs. This is the same program created and updated by ITMU. We advertise it to run for 3 days. It ends at 10:00am on the 3rd day. This way we catch laptop users and people who take a short week. The third program is advertised starting on the 2nd day. This is a reboot notice. We don’t actually reboot machines, instead we let the customer do it. The reboot notice checks the registry to see if the customer has a patch pending reboot. Patches that quit with exit code 3010 will create a temporary reg key for that update: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\ CurrentVersion\WindowsUpdate\AutoUpdate\RebootRequired\DWORD value of each update ID that requires a reboot and sets the value to 1. If we don’t find a key = 1 then the reboot notice is suppressed. If there is valid reboot key, then we display a notice that alerts the customer. The customer may delay the reboot by clicking okay, but not kill the notice. It will pop up again later, until a reboot happens, or the advertisement expires, 8 hours later.
Get the script from Microsoft here:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wua_sdk/wua/using_wua_to_scan_for_updates_offline.asp

We had 4 different collections to setup of all 3 package advertisements for. The first one went out to the entire systems group. It was one last chance to find a problem before hitting the rest of the company, so all of its packages are advertised one day earlier. The second group was our normal internal lan. Known as the Business Network, it hits all our standard operational departments (marketing, accounting, contracts, etc.). The third group was our developer network. Our developers are currently given a choice about having SMS update their system. The ones who opt out must be able to provide a valid reason, so we separated the entire network to make administration a little easier when dealing with the rest of the company. The last network was our RO’s (Regional Offices). We’ll probably roll them into the business network eventually, but for now they are separate.

We did hit an issue with the RO’s, the old admin had a 3 hour time difference for some parts of the patches but not for all. We spent an hour debating if SMS used the Parent Site time, the Child Site time, the DP time, or the client time. While we were certain that most machines in the lan were joined to the domain and therefore had the right time, we couldn’t be as sure about the RO’s. Also, if all the RO’s are in one collection and we start something at 8:00 am our time (Pacific), then it really starts at 11:00 eastern. If we start it at 5:00 am our time then how does that affect things that are expected to run until 5:00pm since they would now be ending at 2:00pm. At one point Admin no. 2 thought that checking the Greenwich Mean Time checkbox would be a good idea. We eventually convinced him that it wasn’t. We finally settled on running the notices at the earlier adjusted time, but running the patch at our local 11:00pm time, which was 8:00pm eastern. Anybody still working at 8:00 and noticing a slowdown should just call it quits for the day.

Day Four: SMS success!

Maybe SMS success is an oxymoron. My current measure of success is not breaking the existing infrastructure. Deploying a package is just extra icing on the cake.

One thing I forgot to mention is that just before we left on Friday we did create a test collection with two test machines. Then we created the test advertisement and crossed our fingers.

Monday was a holiday for us. So today was our first chance to see if the patch deployed. I came in, checked my test machine and there was our custom made post patch reboot message. Firing up the SMS console revealed that the packages ran successfully (not that I expected anything less).

The best way (that I’ve found) to figure out if your packages ran on a test group is to open up the console, expand the Site Database>System Status>Advertisement Status. Select the advertisement you’re interested in checking. Then in the right pane you can see the how many clients received the package, how many started it, success, failures, etc. If you want to get really detailed you can right click on your site>show messages>all. I usually only need to look at the last day or two but you can go back further if you need to. The description column will report back every action that was taken and the results of that action. I think it’s better then any of the reports I’ve found.

Since it was the first day of the week, it was time for the meetings to start. First meeting: Discuss the implementation of SMS in one of our classrooms. The Sys admin for the classrooms manages 5 classrooms of 16 machines each. They are firewalled off from the rest of the company, but holes can be opened up for specific needs. He’s still using ghost to re-image the machines before every class. The problem is, he has 16 different images and some of them fall behind in patch levels. The old admin started a project to get SMS implemented in these classrooms. After hunting around for a few days we found the servers. He setup a child site and a separate machine for dp. We figured the child site is inside our internal firewall and the dp is on the classroom side of the firewall. At this point we assume the correct ports have been opened, guess we’ll find out. A little bit of discussion and we determine that he can automatically run the sms client install after ghosting. Now things get interesting. How do we schedule advertisements such that, the machines are kept up-to-date but no patching takes place while a class is in session. It occurs to me that if he can run a command line automatically after patching then we have a patch solution in place already, currently in use by our new computer deployment team. They use a combination of a WSUS (Windows Server Update Services) Server and a script that updates the Automatic Updates client, forces all the settings, connects to the WSUS server, updates, reboots, logins and continues until the machine is patched.

WSUS? Isn’t that SMS blasphemy? Well hold on before you crucify me. Remember all we are really interested in, is patching classroom machines that get re-imaged several times a week. The software is already installed on the images and we are not patching non-Microsoft products. Plus this solves the scheduling problem, since the patching takes place automatically, immediately after imaging. It’s the right tool for this job.

The idea is well received by the other admins. In the end we agree that SMS is still important to implement, we want the reporting and we see the benefit of creating software packages for the future, and one day we may switch him from Ghost to OSD. Plus it buys us time for him to get the images updated with the client.

Meeting Two will have to wait until my next post.

Day Three: Reality check

Did I say SMS administration wasn’t a full time job and that ITMU was the end all savior answer to our SMS prayers? Time for that reality check.

So here’s what actually happened on Friday. I setup a meeting with admin 2 and 3. I bring my laptop and a projector so we can all see what’s going on. Combined with the patch update document left by the last admin and a little knowledge based on my readings about ITMU, we figured an hour was plenty of time to setup the patch advertisement. Just to be sure I blocked out an hour and half for the meeting. On our agenda: update the existing Microsoft Update patch package to include the January patches, and advertise it to a test group. The patches themselves had already been tested and approved for our environment. Our job was to simply integrate it into SMS and start the monthly company patching process.

Here’s Microsoft’s diagram on how this should work: https://www.microsoft.com/smserver/techinfo/administration/20/using/suspackhowto.mspx

So looking at the document left by the old admin and dated July 2006, we think we have a pretty good handle on how to start this process. Step 1> Right click on the correct advertisement and choose the option to distribute software updates. Well which advertisement is the correct one? Certainly not the one for the entire company. But, there doesn’t appear to be a good test advertisement setup. How can this be? Obviously, something is wrong with this document and we’re only on step one. Okay, no problem. We are three very bright admins (there are 30 Systems Administrators at this company), and the each of the three of us would definitely rank within the top five category for our company. Surely we can figure this out and move on. We start by exploring the options available for every type of object. Strangely, distribute software updates is available via packages, collections, and advertisements. Well, we know we want to update the package, so let’s proceed from there.

Step two: Pick the correct scanning tool (the screenshot show MBSA). We’ll that doesn’t strike us as correct. The DSUW has Microsoft Updates already listed, and based on what comes up in the rest of the wizard (yes we poked around), it appears to default to the values chosen the last time it was run. Additionally, Microsoft’s ITMU documents also appear to use Microsoft Updates. Keep in mind, our company has had SMS in place for 4 years now. The list of scanning tools available include MBSA, Windows update, Office update, and Microsoft Update. Our best guess, Microsoft Update supersedes all the other tools since it combines the features of Windows Update and Office Updates in order to Update both in one shot. This conclusion was reached based on our knowledge of the difference between Windows Update, Office Update, and Microsoft Update’s respective websites.

Moving on to step three: Pick the Q number of the patches to include in your package. Our understanding at this point is that the sync tool has already run and these patches should already be in the package source. Click a couple of corresponding checkboxes and move on. How wrong we were. Since the patch test group had already downloaded the patches and tested them, I know the network share to look at and find the appropriate patches. They are kept in the share and organized by year and then by Microsoft Bulletin number. Example is \\share\2007\MS07-002 . This folder refers to this bulletin: http://www.microsoft.com/technet/security/Bulletin/MS07-002.mspx

As you can see in the bulletin the Office 2000 version of the patch is KB925524. The network share lists each file that was downloaded and the first file in the list is office2000-kb925524-fullfile-enu.exe. Cool, let’s type that into the filter and check the box! Only problem is, 925524 doesn’t return any results. How can this be? Did the sync tool not run? We need to check things out. So we spend the next hour looking for reports, queries, advertisements status queries, checking package source folders etc. As best as we can tell, the sync tool ran on 01/09/07, aka patch Tuesday. So where in the world is 925524? Shouldn’t it be in there? Can Microsoft make this any more difficult? Why isn’t this working right? I’m sure some of the SMS veterans and Security folks would get a real kick out of this train of thought. An hour or so later it’s time to admit defeat and move on to the next one. Excel 2002/xp number KB925523, comes up without an issue. Check a bunch of boxes and we feel pretty damm good. So what happened to 925524. Let’s poke around some more. It’s time to actually read the bulletin. Look at that, MS07-002 has a number next to it: 927198. Maybe that’s the Q number we should be typing in? At this point, I admit we’re lost. I type in every number I can find, check the boxes that come up and ignore the ones that don't. Hopefully the old admin will answer our emails before we deploy to the company.

The rest of the process pretty much proceeds as expected. Updates get downloaded. DP’s get updated. We choose not to update the collections and advertisements, because we want to do it ourselves and not screw up the company (yet).

The hour long meeting only took 4 hours! This SMS stuff is easy as pie.

Day Two: The SMS saga continues

It's my day off, but I want to go over some of the things that happened yesterday.

I confess. I do have some rudimentary understanding of how SMS works. Before the old Admin left I had worked with him on creating a package for deploying Microsoft Project 2003. Also, as I’ve said before, I did most of the implementation of the OSD Feature Pack. Just to give you an idea of where I’m coming from, since we currently do our deployments using the OSD deployment CD, there are no collections or advertisements involved. Also, Image packages are slightly different than regular packages. Not much, but enough so that Microsoft has made it a different object within SMS.

I also did find some of the documentation left behind by the old admin. As I alluded to in my first post, this documentation was buried in a sea of files that dated back to 2003. Many of the processes and procedures that were documented were no longer relevant. However, being the clever SMS admin that I now am, I sorted the documents by date and started working my way backwards from newest to oldest. After 4 hours I had accomplished the task of putting a copy of the current and relevant documents in another network share, which I shared with newly minted SMS admin 2 and 3.

What little jewels did I receive for my hard work? Well besides a pat on the back there was an up-to-date copy of the current SMS Hierarchy. Our site has close to 20 SMS servers! Most are actual site servers with several primary and a couple of child sites. Some are simply distribution points. Is this the best setup? Only time will tell.

In addition to the SMS Hierarchy, I managed to salvage a copy of our implementation of the Microsoft Patching procedure. It had screen shots and detailed step-by-step instructions! Everything you would need to administer Microsoft Patches via SMS. The old admin has actually installed ITMU (Inventory Tool for Microsoft Updates). After a little reading up on ITMU, this little gem was going to be the answer to all our prayers. ITMU will sync all the patches available on Microsoft update with the SMS server using a designated Syncing machine. Then it updates the existing Microsoft Updates Tool package with the new patches. After that it will allow you to test the patches on a pre-production environment. Finally, it will allows you to deploy the updates to the rest of your machines. Life was looking pretty sweet. SMS isn’t really a full time job is it?

Next time: Microsoft best practices and reality clash.

Day One: The SMS Admin has left the building

The SMS Admin has left the company. There are no useful documents. If your lucky there are some buried in the sea of network shares, but most are outdated or poorly written. It’s Mission Impossible. Your mission, should you choose to accept it, is to take over SMS and all of its associated projects and duties without bringing down the company. No this isn’t really optional, but it’s the position I found myself in today, and with patch Tuesday already 3 days old the task wasn’t going to disappear to the bottom of my to-do list.

So what does one do when the reins of SMS have been handed over and the old admin isn’t around to ask questions? I had a little experience with SMS since I helped implement the OSD feature pack. One of the other Admins had experience since he helped install it (3 years ago). Admin number 3 attended a couple of classes, but really didn’t have a clue regarding SMS. It was crunch time and today was day one.

Step one: Install the SMS console. This can be done on any machine and is used to connect to the SMS site server and DP. You can cheat and use the console installed on the server, but who wants to do that? It’s on the SMS CD but in the real world your SMS server is version 2003 and SP2 is installed. The CD won’t help since the latest version is needed which you can get direct from Microsoft: http://www.microsoft.com/downloads/details.aspx?familyid=37B20B4B-DFEC-464D-908B-5D783E2370D3&displaylang=en

Run SMS2003SP2.exe /x

Autorun.exe will get the client tools installed and contains the latest version. It’s probably backwards compatible. So even if your previous admin wasn’t up to par, you can connect to the site server and administer SMS

Step two: Start the console and connect to your central site server. Use the machine name when prompted not the site code. If you didn’t get console on your local machine you can cheat and RDP to the SMS server. You’ll need admin rights either way.

Step three: Look around and see what the previous admin left.

There are 3 primary terms to get familiar with in SMS. 1- Packages 2- Collections 3- Advertisements.

Packages:

A package is a group of source files necessary to run a particular program. In most cases you want the setup program files. It also contains the command line to run. A typical command line is msiexec /I foo.msi /qb. Go ahead and make one. You’ll be asked to update the Distribution Points with it. Nothing is going to the clients so feel free. A good harmless one has no source files just a command line: notepad.exe. Just like start>run>notepad.exe this will cause notepad to pop up. If you choose the option to run it under the system account instead of the local user account then nobody will see it, but it should appear in task manager when the all users option is checked.

Collections:

Collections are simply a group of computers. It can have one or many computers in it. Look for an existing test collection with computers you are familiar with. Better yet, create a new one with one of your test machines in it. You’re not gonna screw anything up by creating one. Don't mess with any of the existing collections some of them are created with a SQL query.

Advertisements:

Advertisements tie Collections and Packages together. This is where you can do damage. Creating an Advert involves telling SMS which packages should go to which collections and when. The wizard is pretty self explanatory, just make sure you pick your notepad package and the collection is your test collection.

Congrats you just did your first SMS deployment on the test server and didn’t bring down the company. (you weren’t practicing on the production server were you?). I’d call this a successful mission. Coming up in Day 2: What the hell is going on with the SMS environment?