|
|
Hello all.... I have seen this topic posted several times but it looks
like most of them were never answered or abandoned some time ago, so I
thought I would post a new thread. We have 150+ Windows Servers that
we patch in groups of 25-30 whenever updates are released by
Microsoft. We are currently using WSUS 2.0 and GPO's to automate the
patching process and scheduling of installs/reboots for these servers.
Over the past several months, we have seen that 10-15 servers in each
of these groups will hang at the "Windows is shutting down..." screen
after applying updates and attempting to auto-reboot. The affected
servers are random at best as far as we can tell (they are usually
different each patch interval).
We have antivirus scanning policies in place to prevent real-time
scanning of the following directories on every server:
C:\Pagefile.sys
C:\System Volume Information All files and sub-directories
C:\Windows\Microsoft.Net All files and sub-directories
C:\Windows\SoftwareDistribution All files and sub-directories
C:\Windows\System32\config All files and sub-directories
C:\Windows\System32\CatRoot2 All files and sub-directories
We are currently using CA eTrust Antivirus, and these problems started
occurring after this software had been in place for a couple of
years.
We have developed a means (sort of) of being able to detect which
servers are in this "stuck" state by looking at our WSUS reports on
Monday mornings after a scheduled Saturday patching. Any servers which
still appear to have "needed" updates we attempt to RDP into. When we
do that on these servers, we receive only a gray screen and never any
logon windows. However, as soon as we touch the server via KVM or
Avocent, it will instantly complete the shut down process and reboot
normally. But if we don't do this, they will remain in this "stuck"
state indefinitely.
We have opened a case with Microsoft, but they have not been able to
determine a cause and/or solution as of yet. They recommended that we
disable all screen savers on our servers as an attempted resolution,
but there we run into a different problem. On a known-good GPO,
whenever we attempt to turn on the settings to disable screen savers
on our servers the settings will never take effect. It's as if the
screen saver settings portion of the GPO are non-existent. Apparently
this is another documented issue that I've seen several posts on with
no resolution.
If anyone has any thoughts and or information on how to resolve the
servers that get stuck at "Windows is shutting down..." issue, it
would be immensely appreciated. Thanks!
|
|
"David in Kansas City" <david.fike[ at ]gmail.com> wrote in message
news:1182179887.313066.167090[ at ]g4g2000hsf.googlegroups.com...
[Quoted Text] > Hello all.... I have seen this topic posted several times but it looks
> like most of them were never answered or abandoned some time ago, so I
> thought I would post a new thread. We have 150+ Windows Servers that
> we patch in groups of 25-30 whenever updates are released by
> Microsoft. We are currently using WSUS 2.0 and GPO's to automate the
> patching process and scheduling of installs/reboots for these servers.
> Over the past several months, we have seen that 10-15 servers in each
> of these groups will hang at the "Windows is shutting down..." screen
> after applying updates and attempting to auto-reboot. The affected
> servers are random at best as far as we can tell (they are usually
> different each patch interval).
Actually, the question has been answered a gazillion times. Those without
resolutions is because they've been unwilling or unable to find the actual
cause, which has *NEVER* been anything associated with the Windows Server
Update Services subsystems.
[a] There's an update for Windows 2000 systems demonstrating this behavior.
Find it and install it.
[b] Otherwise, your AV inquiry is on the right track.
--
Lawrence Garvin, M.S., MCTS, MCP
Independent WSUS Evangelist
MVP-Software Distribution (2005-2007)
https://mvp.support.microsoft.com/profile=30E00990-8F1D-4774-BD62-D095EB07B36E
Everything you need for WSUS is at
http://technet2.microsoft.com/windowsserver/en/technologies/featured/wsus/default.mspx
And, almost everything else is at
http://wsusinfo.onsitechsolutions.com
.....
|
|
Well, all of the affected servers are Windows 2003 Servers - 2003 SP1,
SP2, x64, x32, Enterprise and Standard we have seen affected by this
behavior.
None of our Windows 2000 machines (the very few we have left), have
been affected.
|
|
Is there any update applicable to Windows 2003 Servers?
|
|
"David in Kansas City" <david.fike[ at ]gmail.com> wrote in message
news:1182182996.769217.252380[ at ]o61g2000hsh.googlegroups.com...
[Quoted Text] > Well, all of the affected servers are Windows 2003 Servers - 2003 SP1,
> SP2, x64, x32, Enterprise and Standard we have seen affected by this
> behavior.
>
> None of our Windows 2000 machines (the very few we have left), have
> been affected.
Then it's probably not fixable by a patch. :-)
(at least not a Microsoft patch)
--
Lawrence Garvin, M.S., MCTS, MCP
Independent WSUS Evangelist
MVP-Software Distribution (2005-2007)
https://mvp.support.microsoft.com/profile=30E00990-8F1D-4774-BD62-D095EB07B36E
Everything you need for WSUS is at
http://technet2.microsoft.com/windowsserver/en/technologies/featured/wsus/default.mspx
And, almost everything else is at
http://wsusinfo.onsitechsolutions.com
.....
|
|
|
|
Anyone else have thoughts/ideas? I run through an automated patch
install/reboot on a server last night and it also hung at the "Windows
is Shutting Down..." screen. When I attempted to launch Remote Task
Manager to see what processes were running, it said the RPC Server was
unavailable. This kept me from looking at the Services snap-in and the
Event Viewer snap-in when I attempted to connect to them remotely.
Again, the instant I touched it via our Avocent console (KVM) it shut
down and rebooted normally. But it will stay "stuck" like that
indefinitely until someone touches it via console or KVM.
After it rebooted I looked at the Event Log and this is the last event
recorded prior to the reboot:
"The process winlogon.exe has initiated the restart of computer
SIEDFS0 on behalf of user NT AUTHORITY\SYSTEM for the following
reason: No title for this reason could be found
Reason Code: 0x80020002
Shutdown Type: restart
Comment: "
Microsoft says this might be related to screen savers....??
|
|
David in Kansas City wrote:
[Quoted Text] > Anyone else have thoughts/ideas? I run through an automated patch
> install/reboot on a server last night and it also hung at the "Windows
> is Shutting Down..." screen. [...]
>
> Again, the instant I touched it via our Avocent console (KVM) it shut
> down and rebooted normally. But it will stay "stuck" like that
> indefinitely until someone touches it via console or KVM.
I've seen this infrequently on our Windows XP machines but never on a server.
The usual scapegoat is anti-virus software; is there any installed?
Harry.
|
|
Yes - we are using CA's eTrust Antivirus 8. The problem is that we
have been using this product for a few years now, and have not had
this problem up until several months ago. I do not believe an upgrade
has taken place, but I can't rule that out until I can talk to our
Data Security team. But I do know that as far as scanning exclusions
go, we are excluding the typical Windows system folders from realtime
scans, as well as any Windows Update related folders.
I downloaded a remote task manager utility to see if I could pull up a
list of running tasks on one of these servers when it became "stuck",
but at that point the RPC server service is already stopped, so I
can't connect with it; nor pull up Event Viewer remotely via MMC.
|
|
David in Kansas City wrote:
[Quoted Text] > I downloaded a remote task manager utility to see if I could pull up a
> list of running tasks on one of these servers when it became "stuck",
> but at that point the RPC server service is already stopped, so I
> can't connect with it; nor pull up Event Viewer remotely via MMC.
Gets a bit tricky, but according to the documentation a service that doesn't
accept the shutdown control should continue to run in this scenario. So if you
have any Windows-savvy programmers on hand it should be possible to write a
service that keeps a dialog box open on the Winlogon desktop and launches a
command-line window on demand.
Does the problem happen every time one of the servers is restarted? One of the
reasons it's been hard for me to diagnose what's going on is that it only occurs
very rarely.
Harry.
|
|
It doesn't appear each time a server is restarted. We only see the
issue when we activate our GPO to download and install updates
automatically. When the auto-install is done, the server tries to
restart but gets "stuck" until we touch it with a KVM/console. It's
always a different combination of servers.... some are good one week,
next week they get stuck
|
|
David,
An organization I am working with is experiencing the same issue with
both windows 2003 and windows 2000 boxes. We have been trying various
ways to replicate the issue but cannot seem to find a way to make it
happen consistently. Microsoft recommended we update the WSUS client
to version 3 or disable the screen saver. Since we can't consistently
reproduce the problem we can not test the solutions they have provided
us. Have you had any luck and getting the problem to appear
consistently? In our case usually a reboot fixes the problem and we
can't get it to reoccur.
|
|
We have not been able to reproduce the issue consistently. As best as
I can tell, it appears to be random. Microsoft PSS also suggested that
we update to the 3.0 client as well as disable screen savers. I
manually modified the Local Security Policy on 45 servers this past
weekend to disable screen savers, and we still saw the issue on a
handful of them.
As I mentioned before, it seems to be pretty random and is occuring on
Windows 2000 Server, Windows 2003 SP1, Windows 2003 SP2, Windows 2003
x64, as well as both Standard and Enterprise versions of 2003.
I haven't come up with a way to deploy the AU 3.0 client to all of our
servers, but since the screen saver disabling doesn't appear to have
any effect this will likely be my next move. Our next patch window
won't occur until July 10th, so if I can't find a way to auto-deploy
the 3.0 client I will manually install it on our set of servers that
will be patched first following the next "Patch Tuesday" release. I'll
keep you posted on the results of that. If you happen to notice
anything on your end as well, please let me know! Thanks!
|
|
On Jun 27, 12:44 pm, David in Kansas City <david.f...[ at ]gmail.com>
wrote:
[Quoted Text] > We have not been able to reproduce the issue consistently. As best as
> I can tell, it appears to be random. Microsoft PSS also suggested that
> we update to the 3.0 client as well as disable screen savers. I
> manually modified the Local Security Policy on 45 servers this past
> weekend to disable screen savers, and we still saw the issue on a
> handful of them.
Have you upgraded the firmware on your KVM devices? Try that and the
reboot the servers once and see what happens.
>
> As I mentioned before, it seems to be pretty random and is occuring on
> Windows 2000 Server, Windows 2003 SP1, Windows 2003 SP2, Windows 2003
> x64, as well as both Standard and Enterprise versions of 2003.
>
> I haven't come up with a way to deploy the AU 3.0 client to all of our
> servers, but since the screen saver disabling doesn't appear to have
> any effect this will likely be my next move. Our next patch window
> won't occur until July 10th, so if I can't find a way to auto-deploy
> the 3.0 client I will manually install it on our set of servers that
> will be patched first following the next "Patch Tuesday" release. I'll
> keep you posted on the results of that. If you happen to notice
> anything on your end as well, please let me know! Thanks!
|
|
Gus - I originally thought of doing that, but then remembered that we
are seeing this issue on VM's as well as physical servers; so I don't
think it is a KVM related issue. However, the same problem exists on
the VM's: if we attempt to RDP into a "stuck" server, all we get is a
blank gray screen. We have to open up VMWare Virtual Infrastructure
Client, navigate to the VM in question, and click the 'Console' tab.
As soon as we touch the server using this 'Console' tab, the server
will complete its shutdown and reboot.
|
|
On Jul 2, 11:41 am, David in Kansas City <david.f...[ at ]gmail.com> wrote:
[Quoted Text] > Gus - I originally thought of doing that, but then remembered that we
> are seeing this issue on VM's as well as physical servers; so I don't
> think it is a KVM related issue. However, the same problem exists on
> the VM's: if we attempt to RDP into a "stuck" server, all we get is a
> blank gray screen. We have to open up VMWare Virtual Infrastructure
> Client, navigate to the VM in question, and click the 'Console' tab.
> As soon as we touch the server using this 'Console' tab, the server
> will complete its shutdown and reboot.
We have also found that disabiling the TCP Offload Engine (TOE) on the
HP servers Intel NIC has cleaned up a lot of issues (including the
grey screen) for us since moving to SP2. I can always get in through
the iLO to finish the reboot when needed.
|
|
I appreciate the info Gus and will look into that.... outside of our
VM servers, all of our hardware is Dell PowerEdge 2600 - 2950 using
Broadcom NetExtreme NICs. I will look into the TOE disabling to see if
we even have that installed as an option with our NICs.
|
|
|