Broken Agent commands

  • 28 September 2019
  • 13 replies
  • 637 views

Userlevel 2
Badge +4
As of lately a majority of my endpoints don't accept agent commands, it says they either elapsed, executed (but is never executed on the endpoints) or not received. Sometimes it can take days or weeks for the endpoints to show executed even though they check in every day. Changing policy is instant. After a remote session and submit logs to Webroot, Webroot says there is no fix for this at this time.



Does anyone else experience this issue?

13 replies

Userlevel 6
Badge +26
Thanks @TripleHelix - i don't monitor all of these threads as closely as I should, so thanks for pinging me. @NicCrockett - many key Webrooters are all still here and not going anywhere. 8-)

On to several of your issues/observations. I'll try and cover what I can and provide some insight and/or background on these situations and what will be happening.

NOTE: the current console is pretty much locked for a while in a maintenance mode (This is not official statement, just not much will change with it unless it's "broken")... the reason I state this is, because there is a major initiative to completely redesign/rebuild/release a completely redone console (and backend infrastructure) which should addresses many of these issues. That initiative began late last year and is still in the works even when the merger. That is targeted to be released early next year and some of the backend architectural stuff is being updated in the backend now.

Agent commands:
> Old DOS commands and "Download Run" were pretty much crippled. Initially they were simply disabled and were reintroduced to ONLY support domains, scripts and/or applications that are supported by Webroot, so any general purpose CLI command will fail and is not supported. It had limited support prior, anything not allowed by Webroot will not be supported.
> Current agent commands in the current UI. These are supported by our backend server systems, which are in flux and it's not always clear on when the changes take place, so even us engineers internally we are not always aware of all the details and we see many of the same things you observe and we scream loudly about them. However, be aware that todays server infrastructure is not the same as it was 4 years ago. All unseen or visible to any user, but the agent checks in very differently today than it did a few years ago. There is a lot of work being done on that front and support isn't always aware, so they provide work arounds. Not ideal, but they're focus is to help. Using unmanaged policy and local changes is not good, but it does seem to work.

NOTE: We all communicate heavily with dev team(s) about this and there are many dev/ops teams working to insure the hit/misses are minimized. I wish I had a solid answer, but it's being looked into.

> Overrides:
To clarify not all overrides are handled the same, so let me explain.
  • MD5/hash overrides are instantly available and are stored in the threat console, so agents have immediate access to them when applied through your override configuration in the admin console.
  • Folder/path overrides - are dependent on the agent checking in and pulling down a local file stored in WRDATA folder. If check in is problematic, then this file will not get updated in a timely fashion. OVR file holds that data and it's all encrypted except the udpate string at the top, so that will let you know how recent it's been modified.
>Scanning:
  • Scheduled scans will never align with the time stamp you assign unless you turn off "random" in the scan settings, which is on by default. So, you will never see a scan at 10am every day. It gets even more messy when machines are off, but with servers, it could be random, plus/minus from the schedule. If you want to see consistency, then turn that off and keep an eye out to see if it gets closer to the time you desire.
  • Scheduled scans may or may not report to the console consistently and that's being looked into. When the agent scans, it doesn't not talk directly to your console. It reports to the middleware server infrastructure, which then updates your console. To verify daily scans, spot check the local log file, as the agent has it's schedule local and is probably scanning every day as expected, but may or may not report timely in the console. Again, backend changes could be causing this, but i don't know directly.
  • Scanning in general is becoming less relevant. I don't want to get into a philosophy discussion, but real time shields and other shields are WAY more important than daily scans. (This is my opinion with many years in this space. The likelihood of finding something malicious at a given time (5 minute window) at any given time of the day is like less than 1% probable. It's only real job is to verify any previous threats to insure they're gone and clean, so just expressing a position. 😎

As the new console starts coming along and closer to release, hopefully we'll be getting comms out to the community, so stay tuned and watch for those notifications.

Hope this helps.
Shane
Userlevel 7
Badge +63
@NicCrockett sorry can't help either but we can ping some Webroot Staff members to see if they can help? @cbullas @BruceR @JGiffard @coscooper

HTH!
Userlevel 7
Badge +28
Thanks for your input and letting me know Lara left. I didn't realize she had left the company, although I'm not surprised. Webroot lost a dedicated and competent employee. No offense @freydrew, I think you are also dedicated and competent. 😉

If you're going to test, you don't need Windows 10. The Agent Commands are sent from the GSM web console. You login to the website and set your site up with policies, groups, etc. From there you can click one or more Endpoints to send Agent Commands to. Based on the policies you set-up, there are certain policies that aren't being followed based on @dudeonthebayou and my research.

  1. When each Endpoint checks in with the GSM server for things like these Agent Commands and other things like changes to polices and overrides. Yet another problem I found at the beginning of the year was the overrides database doesn't update properly either. The issue was marked solved, but it was never solved.
  2. Another is when the Endpoint runs a scheduled scan. As you can see in my previous picture, it is both not following the schedule and missing multiple scans on an Endpoint. On top of that, Webroot has said for 4+ years that alerts to warn us of this are coming. You can't test what still doesn't exist.
If all that doesn't bore you to death and you want to know more about the issues I've faced and continue to because of supports "Just Deal With It" mentality, here's some not so light reading: 😂

  1. Webroot House Cleaning - Potentially fatal to a PC.
  2. Uninstalling Webroot - Major issue for admins, like myself.
  3. Agent Version Push Command - admins need when an Endpoint still hasn't received the next version 3 months after release.
  4. Primary Browser Doesn't Work - Shows that reporting can't be trusted.
  5. Reports to Email in PDF - Not a big deal, but would be useful.
Feel free to add comments here or to any of the threads I linked to. If any of you find it worthy of a vote, please do so. However, I don't want to hijack @dudeonthebayou's thread, so let's hope someone can find a fix for them. I just thought it worth mentioning that the Webroot Business product has many massive issues, and these are just the ones from me. Plenty of others have posted questions and ideas as well.

Sincerely,
NicCrockett
Userlevel 7
Badge +62
Sorry, I really can't add anything because I never ran the Business WSA. @Ssherjj I still have that keycode that they gave us. I never used it. I wonder if it's still good. I'll give it a try after I upgrade to Win 10 in the near future.

Yes PTD, I asked Drew for a renewal of the keycode 2 years back or more and I did not ask for it again. Otherwise I could help @NicCrockett test these Commands? But I am sure Nic has it figured out that there is a problem/s...

Upgrading to Win 10 sounds like a great idea!!😊
Userlevel 7
Sorry, I really can't add anything because I never ran the Business WSA. @Ssherjj I still have that keycode that they gave us. I never used it. I wonder if it's still good. I'll give it a try after I upgrade to Win 10 in the near future.
Userlevel 7
Badge +62
Hi @NicCrockett & @dudeonthebayou,

Sorry I cannot be much help here. I used to run the Beta Business version of Webroot but my Beta Keycode has expired and I reluctantly decided not to ask to add more time to that Beta Business Keycode.

I do agree that Support should be looking into these issues in question. Especially when it concerns a Business, which perhaps @freydrew should be on top of this with some answers. He is the only one left here at Webroot who is the Administrator/Management of this Forum. Now that Lara has left the company. 🙁 So hopefully this can be resolved quickly. As Drew can contact the Business Support help when needed.

@dudeonthebayou Maybe this here will help? Or here?

Thank you @NicCrockett for the ping and I wish I had more to add to this sad situation.😖
Userlevel 2
Badge +4
@NicCrockett now that you are saying it I have noticed occasional gaps in scans on a Windows 2012 server a client has.

Thank you for your assistance.
Userlevel 7
Badge +28
Congrats @dudeonthebayou, you've found a bug in the software. I understand why Webroot Support would suggest you try the "Unmanaged" policy. However, suggesting that it's a solution for a business is ludicrous. Unfortunately, I don't have a solution for you. I'm sorry, but I'm not a Webroot employee, let alone one of their developers. However, let's see if we can start a conversation and get some answers, because I'm seeing even worse issues then you are. So, let's start with what I found.

I ran the Scan Agent Command on several of my Endpoints to see what happened. I specifically chose these Endpoints to have a variety of Operating Systems, Physical Endpoints, and Virtual Endpoints. Sorry, all my Endpoints are Windows, so I could only test different versions of Windows. Here's the results of my testing. Sorry for blurring out the hostnames, however I annotated the Operating System. If it has VM at the end, that means Virtual Endpoint, otherwise it's a Physical Endpoint. You'll notice that only 2 Executed. My Endpoints are set via policy to refresh every 15 minutes and I waited 3 hours before ending the test and writing this. On top of that, the Windows 10 Endpoint I refreshed the configuration manually a few minutes after submitting the Scan Agent Command. It was the first Endpoint I attempted the test on and it failed, even with a refresh. I also saved a Scan Log from the Windows 10 Endpoint and it only had a record of scanning at around noon, not 1:49 PM when I sent the Scan Agent Command. You're probably wondering about the red arrow on the one that Executed. Read the next section for why I point that out and why this is a bigger issue than you may have realized.



While checking the Endpoints that I attempted scans on, I looked at the Windows Server 2012 R2 Data Center Virtual Endpoint that Webroot's Command Log says was Executed, red arrow above. The image below is the Scan History for that Endpoint. All my Endpoints have a policy set to run a daily scan at noon if they are only turned on during the workday or at 10 PM if they are always on, i.e. a server. This Endpoint is a VM that is an always on server and is part of a policy that should run a daily scan at 10 PM. This Endpoint was added on April 25th, 2017 and this policy with this schedule hasn't changed since before that. As you can see in the Scan History list below, I've placed 3 red lines that designate large gaps where no scans took place and you can also see that the scans after August 26th, 2019 aren't following the policies set schedule. At a quick glance, I found at least one other Endpoint that this is happening to. The unfortunate thing is, I posted this idea over 4 years ago that they put in an alert for this very problem. If you want to read that catastrophe check out the first idea I submitted and the second time I had to resubmit it.



All that to say, congrats on finding a bug and we know it's worse than you first suspected. Let's see if any other users that have been around for awhile know anything. If they don't, maybe a Webroot Community Admin can either shed some light or pull in someone who can. Please remember that I'm reaching out to these people for help, but they may not have an answer. Let's hope someone can help, but we may have to take this back to Webroot Support.

@Ssherjj, @TripleHelix, @Jasper_The_Rasper, @Muddy7, @LLiddell, and @freydrew. I realize this might not be your territory, but I was wondering if anyone could help out @dudeonthebayou with their problem. It would also be nice if someone could shed some light on problems that I've been seeing for over 4 years. Webroot's official response to their business clients cannot be "Just Deal With It". So, I'm guessing there is a reason these issues are occurring, have persisted for so many years, and improvements to help us know when an issue occurs have been consistently delayed by years. Again, I realize not all of you are Webroot Staff, but I included you because you might know something I don't. I respect all of your opinions and consider you experts, including you @ProTruckDriver. That is why I'm asking for everyone's help. Thanks for any help you can provide!

Sincerely,
NicCrockett
Userlevel 2
Badge +4
The response i received said "There unfortunately no permanent solution available at this time in regards to Agent Commands and their completion on the endpoint(s). Until this is created and implemented you'll need to move the endpoint(s) to an 'Unmanaged' policy and apply any desired changes locally."

If you have 50-60 endpoints then manually changing the key can be a time consuming job.
Userlevel 7
Badge +28
I'm sorry for the confusion! I was way off base on what your issue was. I'm sorry, I didn't even think you would be talking about Agent Commands, as they are basic functions of the GSM. I'm running some tests right now and will get back to you as soon as I can. However, I can tell you that support shouldn't have given you a response like that. I'll let you know what I find.

On a side note for anyone wondering. The ability to run commands like I was talking about, still can't be done because it's still removed from the GSM.

Sincerely,
NicCrockett
Userlevel 2
Badge +4
It's commands like, scan. cleanup, reboot, uninstall, change key etc.

i don't hate Webroot, i just want the software to work as intended.
I was working on moving endpoints to the new console interface
and i noticed it when the endpoints would not change key within
a few days. Just a lot of work manually change the key on endpoints
which is not within "reach" so to speak.
Userlevel 7
Badge +28
Welcome to the Webroot Community @dudeonthebayou.

I'm swamped, but I'll help as best I can. Unfortunately, your issue could be one of multiple potential problems.

First, if memory serves, there was a security issue with Webroot dealing with the admin commands and they quickly and proactively took steps to protect us. Don't worry, the anti-virus and Endpoints were fine. I can't remember the exact details, but one thing it dealt with was being able to run commands. Don't hate Webroot for this, they took necessary steps to keep us safe until a viable solution could be put in place, by shutting down that part of the GSM. I thought this had been fixed, but I wasn't following it closely, so I'm not sure. You might be able to find something about it in the community. If anyone knows what I'm talking about and whether or not it has been fixed, please chime in.

Second, is a potential problem with the commands you are sending to your Endpoints. This is an issue based on the command that you are pushing and the version of Windows running on that Endpoint. Quick example is the DOS command wuauclt /detectnow. It's a simple command that tells a PC to run Windows Update. This command works via Webroot for Windows XP and Windows Server 2003. However, it stopped working for Windows 7 and Windows Server 2012. I can manually type that command in on the Windows 7 PC, but it won't accept it from Webroot. Your description, sounds a lot like this where it takes time, says it executes, but doesn't. Again, not Webroot's fault. Changes to Windows security (UAC) made it impossible for certain commands that Microsoft deems administrative impossible for Webroot to run because it can't act as the administrator.

Just out of curiosity, can you tell me what command(s) you were trying to run? I might be able to help with more info. That's a big might, since I'm not in your environment, but I can try.🤔

Sincerely,
NicCrockett
Userlevel 7
Hello @dudeonthebayou Welcome to the Webroot Community Forum. ☺️

Let me ping @NicCrockett he runs the business WSA. Maybe he can answer your question on seeing the same thing you are seeing.

Reply