For much longer than I care to admit I was struggling to monitor any Windows services using WMI on one of my Windows 2008 Servers. Instead I received the dreaded "Invalid Class" error in the APM info window. This was later confirmed using WBEMTEST. When using WMI Explorer it was clear to see that the "Win32_PerfRawData_PerfProc_Process" class did not exist. 

I opened a case with Solarwinds support and was directed to their WMI support document and related online documentation. Unfortunately none of the information provided proved useful in this circumstance.

I spent a several hours googling around and was able to find several others with my issue, but no one had found a solution. With nowhere else to go, I spent the $250.00 and called Microsoft Support. It took a few days, and at least three Microsoft support engineers later that we finally got to the root of the problem. I knew from the very beginning that this was going to be something stupid, but I never realized just how obscure the problem would be, so I decided to save others my pain and post my resolution in hopes that others might find it useful in the future. 

We went into registry “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\PerfProc\Performance” In the right hand pane we found that “Disable Performance Counters” was set to 1, which means it was disabled. We used “exctrlst.exe” tool, enabled ‘perfproc’ and rebooted. After Rebooting we were able to run the WMI query (SELECT * from Win32_PerfRawData_PerfProc_Process where NAME <> '_Total') using the WBEMTEST tool built into Windows. After verifying with WBEMTEST I confirmed that APM was properly polling my monitored services properly. 

I hope someone else finds this helpful.

I wanted to share the fix that I found for the problems I was having related to getting the hardware monitoring working on our HP ProLiant servers that I covered with SolarWinds under case #343875.

The problem we were having was that we had three different conditions with the hardware monitoring not working within Solarwinds on our HP Proliant servers. (All of them were DL G5/G6/G7/G8 series). Two of these conditions generating an error condition that you could see from within SolarWinds that showed up as Hardware Polling failded: Error 31005 (or Error 31040)

All of the issues are related to the HP Insight Management Agent either not being installed or not upgraded or compatible with the version of HP SIM or the WBEM provider that is installed on the server.

In our case we were on SIM version 7, so we needed to install the HP Insight Management Agent version 9.0.0.0. In some cases we still had version 8.70 installed and that is why it was not working. Once we upgraded to version 9.0.0.0, we no longer had a problem once we followed the re-add process below. Accept all of the default options, there is no need to change any of them.

In other situations we had version 7 of SIM installed, but we had no version of the HP Insight Management Agent installed at all. In these cases we installed the HP Insight Management Agent on the server and followed the re-add process below. - Accept all of the default options, there is no need to change any of them.

Still yet in order situations, we were getting a “Hardware polling failed: Error 31002” – In these cases we have found that it is best to uninstall the HP Insight Management Agent, then install the new version from the command line using the /s (for silent install) /f (for force) – the command would look like >x64.exe /s /f       when installing the x64 bit edition of the HP Insight Management Agent. You will not have to set any options because the forced silent install, installs the agent with the default options.

After getting the proper version of the HP Insight Management Agent downloade and installed, you will need to follow the re-add hardware monitoring process below –

> Click on the server node

> Click Rediscover

> Click Poll Now

> Click List Resources

> Select the Hardware Monitoring option by placing a check mark in the box and then clicking submit to apply

> Clock Poll Now Again

You might have to refresh your browser window a few times depending on how long it takes to get to polling the node again, but once it does your hardware monitoring should show up again.

Caling HP support was pretty useless at helping me get to the bottom of this issue, but I suspect that others will see this problem and be looking for a way to fix it like I did.

I would be happy to answer what questions I can if you have other issues get the HP hardware monitoring to work as I think we have been through most of it over the last couple of days.

Hello All

Just a Quick Question on Group Status and roll up status.

I understand that if i have a group with the adv option for status set to "Show worst status" that if a node goes down the group goes to RED if it goes to Warning the Group goes to WARNING.

How can i if possible effect the group status if an application goes warning / RED effect the group status

Regards

Mark

There have been a few posts and cases recently where people have been running into issue with RPC in some form or other and I just wanted to share some information on RPC itself, some troubleshooting and where to look for more information. With RPC, it has to be said Microsoft are going to be the best source of information and most of the details in this post come from Technet.

Errors and Fixes

Error: “The RPC Server is Unavailable”

This error can occur when the remote Server is unable to receive or respond to RPC Requests.

Verify there is no firewall preventing the Orion host from performing RPC calls. A simple method is to verify with an administrator account that you can connect the the remote computer through Computer Management (compmgmt.msc) or that Windows file sharing is possible.

-When checking the firewall/ACLs also check your anti-virus, some AV systems have firewalls built in.

-Windows RPC will operate over a large range of ports depending on the method of RPC in use, by default port 135 is used and the RPC endpoint mapper then will offload the request to the ports the service is running on.

Sources: http://support.microsoft.com/kb/832017, SolarWinds port requirements.

For SAM RPC monitors the majority will be using Named Pipes.

-It also goes without saying to insure the RPC service is started on the remote computer but you will also need to check the Remote Registry Service.

-Check DNS resolution (and NetBIOS)

-Check Windows File and printer sharing is enabled

Next Steps if above fail:

-Check your Windows System logs

Error: "Access is denied"

This is a pure permissions error, adding the user account being used on the RPC client to the administrators group on the remote computer (the RPC server) will clear this. In newer versions of Windows there is a Performance Monitor Users" group you can use in most cases. If this gives problems you can also look at using SC (http://knowledgebase.solarwinds.com/kb/questions/3301/User+permissions+and+RPC. has more detail.).

Error: "Network Path not found"

This points to permissions again but to an issue on the server itself, http://knowledgebase.solarwinds.com/kb/questions/2500/The+Network+path+was+not+found+error+when+you+browse+for+APM+performance+counters

Error:” The specified object is not found on the system”

This can be as easy as restarting the Remote Registry service. To confirm this will fix try find the counters SAM is polling in perfmon when connecting from the Orion server and another remote server. If they don’t display remotely but do display locally then the problem is with the remote registry service or registry permissions.

Error: “RPC server is too busy to complete this operation”

This is tricky, it is likely resources or just plain load on the server. If you’ve Exchange running on the server with this error then you are going to be looking to get it fixed quickly. http://technet.microsoft.com/en-us/library/cc940522.aspx might help but if you are the admin the link mentions, well... :/

Next steps:

-Often times you may find you get an error code when a RPC monitor fails in SAM, http://msdn.microsoft.com/en-us/library/windows/desktop/ms681386%28v=vs.85%29.aspx can help with these.

-Confirming you can access the performance counters manually can help to make sure you aren't dealing with a performance counter issue, just open perfmon and add a counter for whatever you are monitoring. If this fails http://support.microsoft.com/kb/2554336 might help.

For more in depth troubleshooting:

http://social.technet.microsoft.com/wiki/contents/articles/4494.troubleshooting-the-rpc-server-is-unavailable.aspx (This is the best article out there for RPC troubleshooting imo.)

http://lab.technet.microsoft.com/en-us/magazine/cc138001

http://support.microsoft.com/kb/831051

http://support.microsoft.com/kb/832919

http://support.microsoft.com/kb/197814

If the above don't help, then you are looking at a Microsoft Support case or at improving your understanding of RPC so you can troubleshoot further.

General Information on RPC

RPC (Remote Procedure Call) is an open standard, the OSI and DCE were involved in the protocol definitions and RFC 5531 gives the protocol specifications. RPC is a client/server model implementing a method of initiating a procedure on one device (client) and having the procedure process and return information from another device (server).

To do this RPC needs to locate the remote server, connect to it, authenticate and make a request the server can understand and process. The server will then respond, the client will consider the call as completed and continue, processing the information it received.

To locate the remote server RPC relies on DNS or WINS.

To connect to the server RPC will use, TCP, UDP, HTTP, HTTPS, Named Pipes, Winsock or other configured method (I'm pretty sure I don't know them all). RPC itself is totally unconcerned with the network, reliability and routing and it leaves this to the lower layers. This is why you can get RPC over TCP, over UDP, etc.

The client will initially connect to RPC on port 135, the client will be speaking to the endpoint mapper on this port. The endpoint mapper will direct the client to the correct port for the service it is trying to access (the client can and does cache this information). Services can run on any port but are often on the user ports 1025-5000 or 49152-65535 depending on the version and patch level of your server. These ports are often known as the ephermeral ports and some anti-virus systems can limit or restrict this, most will log to the system or application log if they do and most will have a KB on configuring the anti-virus so they don't filter/limit the ports.

For authentication Microsoft RPC uses Kerberos or NTLM authentication. The method is configured through group policy on a whole server level, in general you shouldn't have to adjust the authentication method unless it is changing across your environment.

If you are having any authentication issues in your environment it is likely to affect RPC and if you are having intermittent authentication issue RPC is likely to have them also. Some for the further reading links point to running dcdiag for these sorts of problems in domain environments.

When it comes to permissions needed it depends on what you are using RPC for, http://knowledgebase.solarwinds.com/kb/questions/2500/The+Network+path+was+not+found+error+when+you+browse+for+APM+performance+counters covers the permissions needed for SAM's default monitors.

So, what is RPC used for? Well pretty much everything, RPC is a good technology and much of Windows relies on it. While many services depend on RPC, Exchange, DCs and SQL are probably where people run into RPC errors that cause them the most headaches. Exchange provides performance counters to review how many RPC requests it is getting and how many are outstanding. SAM's default Exchange monitors include these by default (http://technet.microsoft.com/en-us/library/dd335215 has more information on the counters).

Besides Exchange, http://support.microsoft.com/kb/177446 details Microsoft's recommended counters for performance monitoring of RPC and if you are having a problem it can be worth monitoring these along with the Windows Event logs.

References and further reading:

http://technet.microsoft.com/en-us/library/cc738291%28v=ws.10%29.aspx

http://blogs.technet.com/b/networking/archive/2008/10/24/rpc-to-go-v-1.aspx

http://support.microsoft.com/kb/177446

http://support.microsoft.com/kb/325930

http://support.microsoft.com/kb/154596 (RPC with firewalls, if you use this please share your experiences below)

http://technet.microsoft.com/en-us/library/hh875578%28v=ws.10%29.aspx

http://support.microsoft.com/kb/167260

http://blogs.msdn.com/b/bclteam/archive/2006/09/08/746900.aspx

http://technet.microsoft.com/en-us/library/cc779303%28WS.10%29.aspx#BKMK_1

Also http://tools.ietf.org/html/rfc5531 isn't too heavy a read and gives a good over view of the process, worth a look to confirm your understanding of the other links.

Hi All,

New to Orion, and working to get it setup in our environment. Google hasn't helped much with this one. Can anyone suggest the WMI Namespace that I would need to add to look at MSMQ Queue Sizes? I would like to setup alerts if a queue has more than X messages in it.

Thanks,
Chris

I have a host that shows the error message "Unexpected error occured. Invalid class" for all Windows Service Monitors (using the application template Windows 2003-2008 so DTC, Network Connections, Protected Storage and Remote Registry).

I've tried the wmiadap /f command and rebuilt the WMI repository, etc to no avail.

Has anyone else experienced this issue?

We are trying to add a Windows 2012 Hyper-V server to Orion but are having trouble getting WMI configured so that Orion can poll it.  Has anybody else had problems with this or have any suggestions on how to get it working properly?

Thanks in advance for any suggestions!

I have three windows boxes that I'm monitoring via WMI that I cannot seem to pull CPU data for. In the Node Details screen, the cpu chart just says "Data is not available". I'm getting everything else I need from these hosts - disk, memory, interface stats are coming through just fine. Only the CPU data won't come up.

If I run perfmon on the SAM server and connect to the server remotely, I can pull CPU data just fine. Only SAM doesn't seem to see it.

Any hints on what else I can check?

We are having problems getting process monitor to work on our HP-UX boxes.  Have had admin install net-snmp and point to port 1161  (they also run hp-snmp on port 161)  We are receiving all basic system info.  Pronlem is when I run the Process Monitor-SNMP in App Builder in APM in returns no running processes.  running a ps -e command on box shows many process running.   Can anyone assist on what could be the cause and or solution??   config file only contains cummunity string info, allowed ip's (which is our Solarwinds boxes) and pointing agent to port 1161.

HELP PLEASE!!!!

Hi clever people...

has anyone managed to solve the problem of monitoring WMI stuff from APM, when the APM poller sits on the remote site of a firewall tot he device being monitored? if i disable all rules, and make it any:any, then it works but if i have a firewall blocking all but the WMI single port, and have NATs in place, then i am scuppered.

any help would be great!

Solarwinds, what i would ideally need is a remote poller for APM, to sit on the remote site of the firewall and router, and report back into my set of ALX pollers in my management network...

any help?

I have one server that will not return the status of for the services being monitored.  AND, the cpu information is not returned for this node.  We ran the shell command 'get-wmiobject -class win32_service -computer name.domain.local and the service statuses are returned.  The application template used for this server was created using SolarWind's Component Monitor Wizard and the services were picked from the list of currently running services.  So why can the wizard see the services running, but the poller doesn't.  The real time process poller is also failing with the same error.  But, the service control manager works fine.

Is there a monitor or report that will monitor the count of databases on a sql server or cluster? I would like to be able to monitor all sql instances on a cluster or at least be able to find out how many databases are running on a cluster or SQL server without having to open sql manager and counting them.

Thanks in advance if this has been done or in the works.

So I am trying to create an Application view of a group of nodes.  Meaning, I want to see a list of all applications being monitored by nodes in a group.

If I create a summary view, open SAM Application Summary Reports, and check off All Applications Tree, I am able to Preview the view, Edit the item, and use Filter Applications (SWQL) to set up a filter, but I do not know if it is possible to then build a filter based on group membership, instead of something like Caption, or a Custom Property.

Thoughts?

SolarWinds Server & Application Monitor 5.5 has been officially released and is now available for download. All SAM customers under active maintenance can download SAM 5.5 from the customer portal.  For more information on what new features are included in SAM 5.5 please see the following blog postings. You can also  reference the SAM 5.5 release notes for more information on what's included in this release.

This is Ground Control to Sergeant SAM

The Next Level of Server Monitoring - SAM 5.5 Beta 3

SAM 5.5 RC3 - You Got Your Chocolate In My Peanut Butter

Hi,

We were doing MS 2003 server patches the other day and we rcvd 100's of alert emails.  I'm wondering if next month, there is a better way than just turning off (individually) all the alerts prior to doing the maintenance?

Cheers,

archerinvan

Hello,

I am far from being a computer geek and am just very slowly starting to be able to "read" Orion but I need to set up monitoring of MSMQ services on Windows 2008 server so I can set up an alert based on the count of the items in the MSMSQ queue. (Using APM 4.2)

There is actually Application Monitor using Performance Counter Monitor already in place on it but the Performance Counter shows status down with the error Bad input parameter. Category does not exist.

Considering that Performance Counter Monitors read Windows Performance Counter data using Remote Procedure Calls (RPC) instead of Windows Management Instrumentation (WMI).

I am assuming I need to check if it exists in in the root/CIMV2 namespace. I am not quite sure if I need to look just for the variable defined in the Category field of the Performance Counter Monitor or do I need to have more than that in there.

But I may be completely off. Any direction and suggestion you may have  will be, as always, very much appreciated.

Thanks for your time!

Not sure if something has broken in SAM at some point... I am using the 2007/2010 OWA monitor in SAM 5.2.0 SP1 and am seeing the monitor in an unknown state with the reason;

Unexpected error occurred. Failed to deserialize probe settings, probe skipped. Original Exception message is: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel..

If I turn on debugging I see an entry stating:

2013-04-21 13:08:49,845 [STP Pool:752 Thread #0] [C3386] ERROR SolarWinds.APM.Probes.ProbeBase`1 - Failed to deserialize probe settings.

System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

However, the application template is set to ignore CA and CN errors!

Is this affecting anyone else, is it something that has been fixed in a later release, or do I need to raise this as an "undocumented feature"?

Hi,

I am working on creating advanced alert. I am triggering this alert action: log alert to NEtPerfMon Event Log.

Can I set the severity of this alert when it is added to Event Log?

I want to set those alerts to appear with critical severity when they are displayed in Event Log.

Hi clever people...

has anyone managed to solve the problem of monitoring WMI stuff from APM, when the APM poller sits on the remote site of a firewall tot he device being monitored? if i disable all rules, and make it any:any, then it works but if i have a firewall blocking all but the WMI single port, and have NATs in place, then i am scuppered.

any help would be great!

Solarwinds, what i would ideally need is a remote poller for APM, to sit on the remote site of the firewall and router, and report back into my set of ALX pollers in my management network...

any help?

There have been a few posts and cases recently where people have been running into issue with RPC in some form or other and I just wanted to share some information on RPC itself, some troubleshooting and where to look for more information. With RPC, it has to be said Microsoft are going to be the best source of information and most of the details in this post come from Technet.

Errors and Fixes

Error: “The RPC Server is Unavailable”

This error can occur when the remote Server is unable to receive or respond to RPC Requests.

Verify there is no firewall preventing the Orion host from performing RPC calls. A simple method is to verify with an administrator account that you can connect the the remote computer through Computer Management (compmgmt.msc) or that Windows file sharing is possible.

-When checking the firewall/ACLs also check your anti-virus, some AV systems have firewalls built in.

-Windows RPC will operate over a large range of ports depending on the method of RPC in use, by default port 135 is used and the RPC endpoint mapper then will offload the request to the ports the service is running on.

Sources: http://support.microsoft.com/kb/832017, SolarWinds port requirements.

For SAM RPC monitors the majority will be using Named Pipes.

-It also goes without saying to insure the RPC service is started on the remote computer but you will also need to check the Remote Registry Service.

-Check DNS resolution (and NetBIOS)

-Check Windows File and printer sharing is enabled

Next Steps if above fail:

-Check your Windows System logs

Error: "Access is denied"

This is a pure permissions error, adding the user account being used on the RPC client to the administrators group on the remote computer (the RPC server) will clear this. In newer versions of Windows there is a Performance Monitor Users" group you can use in most cases. If this gives problems you can also look at using SC (http://knowledgebase.solarwinds.com/kb/questions/3301/User+permissions+and+RPC. has more detail.).

Error: "Network Path not found"

This points to permissions again but to an issue on the server itself, http://knowledgebase.solarwinds.com/kb/questions/2500/The+Network+path+was+not+found+error+when+you+browse+for+APM+performance+counters

Error:” The specified object is not found on the system”

This can be as easy as restarting the Remote Registry service. To confirm this will fix try find the counters SAM is polling in perfmon when connecting from the Orion server and another remote server. If they don’t display remotely but do display locally then the problem is with the remote registry service or registry permissions.

Error: “RPC server is too busy to complete this operation”

This is tricky, it is likely resources or just plain load on the server. If you’ve Exchange running on the server with this error then you are going to be looking to get it fixed quickly. http://technet.microsoft.com/en-us/library/cc940522.aspx might help but if you are the admin the link mentions, well... :/

Next steps:

-Often times you may find you get an error code when a RPC monitor fails in SAM, http://msdn.microsoft.com/en-us/library/windows/desktop/ms681386%28v=vs.85%29.aspx can help with these.

-Confirming you can access the performance counters manually can help to make sure you aren't dealing with a performance counter issue, just open perfmon and add a counter for whatever you are monitoring. If this fails http://support.microsoft.com/kb/2554336 might help.

For more in depth troubleshooting:

http://social.technet.microsoft.com/wiki/contents/articles/4494.troubleshooting-the-rpc-server-is-unavailable.aspx (This is the best article out there for RPC troubleshooting imo.)

http://lab.technet.microsoft.com/en-us/magazine/cc138001

http://support.microsoft.com/kb/831051

http://support.microsoft.com/kb/832919

http://support.microsoft.com/kb/197814

If the above don't help, then you are looking at a Microsoft Support case or at improving your understanding of RPC so you can troubleshoot further.

General Information on RPC

RPC (Remote Procedure Call) is an open standard, the OSI and DCE were involved in the protocol definitions and RFC 5531 gives the protocol specifications. RPC is a client/server model implementing a method of initiating a procedure on one device (client) and having the procedure process and return information from another device (server).

To do this RPC needs to locate the remote server, connect to it, authenticate and make a request the server can understand and process. The server will then respond, the client will consider the call as completed and continue, processing the information it received.

To locate the remote server RPC relies on DNS or WINS.

To connect to the server RPC will use, TCP, UDP, HTTP, HTTPS, Named Pipes, Winsock or other configured method (I'm pretty sure I don't know them all). RPC itself is totally unconcerned with the network, reliability and routing and it leaves this to the lower layers. This is why you can get RPC over TCP, over UDP, etc.

The client will initially connect to RPC on port 135, the client will be speaking to the endpoint mapper on this port. The endpoint mapper will direct the client to the correct port for the service it is trying to access (the client can and does cache this information). Services can run on any port but are often on the user ports 1025-5000 or 49152-65535 depending on the version and patch level of your server. These ports are often known as the ephermeral ports and some anti-virus systems can limit or restrict this, most will log to the system or application log if they do and most will have a KB on configuring the anti-virus so they don't filter/limit the ports.

For authentication Microsoft RPC uses Kerberos or NTLM authentication. The method is configured through group policy on a whole server level, in general you shouldn't have to adjust the authentication method unless it is changing across your environment.

If you are having any authentication issues in your environment it is likely to affect RPC and if you are having intermittent authentication issue RPC is likely to have them also. Some for the further reading links point to running dcdiag for these sorts of problems in domain environments.

When it comes to permissions needed it depends on what you are using RPC for, http://knowledgebase.solarwinds.com/kb/questions/2500/The+Network+path+was+not+found+error+when+you+browse+for+APM+performance+counters covers the permissions needed for SAM's default monitors.

So, what is RPC used for? Well pretty much everything, RPC is a good technology and much of Windows relies on it. While many services depend on RPC, Exchange, DCs and SQL are probably where people run into RPC errors that cause them the most headaches. Exchange provides performance counters to review how many RPC requests it is getting and how many are outstanding. SAM's default Exchange monitors include these by default (http://technet.microsoft.com/en-us/library/dd335215 has more information on the counters).

Besides Exchange, http://support.microsoft.com/kb/177446 details Microsoft's recommended counters for performance monitoring of RPC and if you are having a problem it can be worth monitoring these along with the Windows Event logs.

References and further reading:

http://technet.microsoft.com/en-us/library/cc738291%28v=ws.10%29.aspx

http://blogs.technet.com/b/networking/archive/2008/10/24/rpc-to-go-v-1.aspx

http://support.microsoft.com/kb/177446

http://support.microsoft.com/kb/325930

http://support.microsoft.com/kb/154596 (RPC with firewalls, if you use this please share your experiences below)

http://technet.microsoft.com/en-us/library/hh875578%28v=ws.10%29.aspx

http://support.microsoft.com/kb/167260

http://blogs.msdn.com/b/bclteam/archive/2006/09/08/746900.aspx

http://technet.microsoft.com/en-us/library/cc779303%28WS.10%29.aspx#BKMK_1

Also http://tools.ietf.org/html/rfc5531 isn't too heavy a read and gives a good over view of the process, worth a look to confirm your understanding of the other links.