Solved

W32.Trojan.Gen. False Positive Fix - April 24



Show first post

289 replies

I have called the number and a ticket was created for me. She said she would send it to escalations in Australia because the US office is closed. I didn't see the note about Sean or Lucas. Ticket number is 85056. I would like to test.

I do want to say after manually cleaning a few computers, they did get flagged again with quarantined files. The machine are XP.
Userlevel 7
Badge +48
@ please call 1 866 254 8400  and if someone from Australia answers, ask for Shawn or Lucas. They are triaging this in our Broomfield office. Please post back here if you have problems and we will troubleshoot. 
Userlevel 7
Badge +35
UPDATE APRIL 25, 2017:  We have a final beta version of the false positive repair utility ready for immediate evaluation. We need five additional customers to participate in our test. If you would like to participate, please call our support team at one of the following numbers:
 
 
Business support phone numbers

US Support (toll free)
1-866-254-8400

Australia Support (toll free)
Australia Support (direct line)
1 800 848 307
+61 (0) 8071 1903

Ireland Support (toll free)
1 800 902 213

UK Support (toll free)
+44 (0) 808 101 7260
Userlevel 1
Appreciate the advice :)

I'd already pushed from the console but certain systems aren't reporting back to the cloud.
I've run wrsa.exe -poll from each of the systems with no success.
 
Not sure what to do with them, they're workstations that are locked down with no access to modify webroot outside of the cloud console (We won't be doing that again I don't think lol)
 
It's been 24+ hours, I'm considering just going into safemode and removing webroot and taking my chances as we need to get these up and running again.
Userlevel 2
@ I ran into a similar issue with 1 machine that would absolutely not giving over the quarantine files (even though the cloud console confirmed that the release had executed). I don't know what exactly I did to fix it but these are the steps that I did take and eventually it worked:
 
1.) Confirmed that the endpoint was set to "unmanaged" in the admin console.
2.) Refreshed the workstation's endpoint.
3.) Confirmed that the user account that I was in was added to the local administrator group (I had a lot of files that were in the C:WindowsSystem ) and UAC was disabled. (I don't know if this matters but worth a shot)
4.) Could confirm that the files were still in the Webroot Quarantine (You should be able to see them in the C:Quarantine or even though the endpoints quarantine)
5.) Right clicked on the workstation's webroot > PC Security (cog) > Block / Allow
6.) Allowed ALL currently blocked files and saved. 
7.) Went back to PC Security > Quarantine > Release
 
It took a few minutes and I checked the folders that the .exe were in, and I could confirm that the programs were back in their place. This was the only one computer that I ran into out of the 200+ that I worked on. Let me know if this works for you. :)
 
 
Userlevel 7
Badge +48
@ Thanks for the heads up. 
By the way - potentially get ready for an influx of angry Australian and New Zealand webroot customers.
 
We had a public holiday yesterday so people who were not monitoring their systems in a way that would have detected this will be walking into this issue now - it is 9.45am Sydney time.
 
From a very specific Australian (and possibly Canadian) perspective we saw Webroot delete IRESS - a very popular Australian share trading & market data application - think an Aussie Bloomberg.
 
We also lost a bunch of our PRTG probes at customer sites because the probes had Webroot installed, which detected the PRTG process as malicious.
 
We also lost HEAT security software for preventing USB use and whitelisting applications (ironically).
 
We also saw an issue where the top level of our portal wasn't showing the infections from some of our customers, leading us to miss some damage that was done until we had customers call in this morning.
 
Stay safe out there ANZACS...
Userlevel 2
@ Do you mean switching the policies to unmanaged or waiting on the command as it sits at "Not yet received"?
 
If it's for switching policies, I have not encountered any issue. Here is what I did in switching policies:
 
1.) Go to Webroot Console
2.) Endpoint Protect > Find the end point > Apply policy to endpoints > Unmanaged
2.) Log into the workstation, right click on the Webroot agent, hit Refresh Configuration
3.) Once policy is confirmed, Righ click Webroot > Open > PC Security (Cog next to it) > Quarantine > Select all that Apply > Restore
4.) This may have a while to release.
5.) Once complete, you can put the end point back into managed mode. (You can also run a verification but by that point, I didn't need to)
 
As for waiting on the commands, it took several hours for commands to be pushed as there is a huge backlog. Last night it took over 8 hours for the backlog to catch up. You can attempt to poll via the command line but I had very little success in this. 
 For 32bit Operating Systems, type:       "C:Program FilesWebrootWRSA.exe" -poll For 64bit Operating Systems, type:       "C:Program Files (x86)WebrootWRSA.exe" -poll  You have to make sure that the endpoint is online or it won't work! Had this happen to several today where the endpoint showed online but the workstation wasn't. I've only encountered 1 issue through all the endpoints I went through ( 200 + ). If you need anything else, feel free to PM me. 🙂
Userlevel 7
Badge +35
Please find the letter that our MSP partners may share with their customers who were affected here:
 
https:///webroot/attachments/webroot/ent2/1451/1/WebrootIncidentResponseLetter.pdf
Userlevel 7
Badge +48
UPDATE: We will be sending an email out to all our MSP partners for them to forward to their customers. It clearly states that Webroot was responsible for this issue, not our Partner MSPs, and reiterates the fact we are working on a comprehensive solution. Please be on the lookout. As a backup, we will post that email here on this thread.  
Userlevel 1
 
Also does anyone know of a way to get agents to unmanaged if console command is not working????
 
This would be beyond valuable. I have had some success getting commands pushed to workstations but a few refuse to push out. Any suggestions would be appreciated.
Userlevel 2
@
 
That wasn't hard thanks for the update. Yes taking a beating because some of us have been trying to help clients through this since yesterday with no sleep, so sometimes frustration gets ahold of us. We appreciate your efforts and if you or who ever would continue with valuable updates like that it would be appreciated.
 
I have tried these steps so far and seem to be having a little success on some machines but have others that simple will not release the quarintined files
 
Posted by @
I know most of you know this but this is what has been working very well for us.
-Switch policy to unmanged in the Dashboard
-on local device, right click and Refresh Dashboard on WR tray icon
-on local device release the quaratined files
-on dashboard run the "Re-verify all file...." Agent command
-on local device, right click and Refresh Dashboard on WR tray icon
-Switch policy to back to the actual policy in the Dashboard
 
If anyone has had any luck getting commands to work from web console I would love to hear input.
 
Also does anyone know of a way to get agents to unmanaged if console command is not working????
 
Userlevel 2
@ Thank you for the update. I know you guys are taking a beating right now so thank you for sticking around with us. 🙂
Userlevel 2
@ I ran into this exact issue on several workstations today. Mostly it was catching our remote agent installer in our %netlogon% on all servers but then it was doubling and even tripling. I hadn't seen it on all of the workstations but I did find it quite odd that it wasn't showing up in the cloud console. 
Userlevel 5
Hi Folks
Just to recap, here is where we are at this point. The best current manual approach to fixing affected machines are the options posted on Webroot Support. There are several different specific techniques on that page that can be used for your critical situations. Its manual, but effective.
We are also continuing work on a "from the cloud" en masse approach to sending agent commands to restore from quarantine for all affected customers. It has proven to be trickier than we anticipated to get it right without downstream issues and test it with both live customers (using volunteers) and internal accounts. More on this as we progress it.
We are also working on a different approach in which you would download a small app that can be pushed to your endpoints that could be kicked off with a command and the app would do the restoration. This approach has not been completed or tested, so no timetable.
Lastly thank you to all on the forum especially those who have generously shared your own successful approaches with your colleagues here. Our support people are available to help of course, but its great to see the community sharing.
More news later.
MIke
I know most of you know this but this is what has been working very well for us.
 
-Switch policy to unmanged in the Dashboard
-on local device, right click and Refresh Dashboard on WR tray icon
-on local device release the quaratined files
-on dashboard run the "Re-verify all file...." Agent command
-on local device, right click and Refresh Dashboard on WR tray icon
-Switch policy to back to the actual policy in the Dashboard
 
 
 
 
 
Userlevel 2
Things have died down on our end. This morning was a little rough but not as bad as it could have been. I spent a greater part of my evening lastnight adding exceptions and then running the quarantine release one-by-one (Beer and ice cream helped). I woke up this morning at 5 AM to check on the progress and most of the backlog had caught up. This gave me hope that the world wasn't ending. It was just going to take time.
 
As someone pointed out, it wasn't so much of a "DO YOU KNOW WHO I AM?" but more of a reaction that the initial fix wasn't a feasible option for an MSP who deals with thousands of endpoints. And then when pointed out that this would not work for an MSP who does have thousands of endpoints, it just felt that the ball was dropped and we were left in the dark. Don't get me wrong, I like Webroot's product. I've worked with them for several years and will most likely continue working with them. We can't fix the past but I would like to know how Webroot plans to correct this so this doesn't occur again. What tools can they provide to us so that we can put a stop (if possible) instead of waiting more than a day. I commend all the Webroot techs and the sales reps that I've spoken with. They've been understanding and I know my day hasn't been as bad as theirs.
 
Anyways, one thing I did notice is that even though the command for releasing the quarantine had executed, we still had to manually remote to the computers. And then there was one odd computer that even though we released the quarantine and never actually released the quaratine. We then discovered that even though it was in an unmanaged mode, all of the .exe were set to be "blocked". Once we unblocked these, everything released and the workstation began working normally. Hope that helps anyone who encounter that issue like we did. :)
 
 
     I've been having the same experience where restoring quarantine from the Cloud portal does nto seem to work. I've had to use the unmanaged polciy and local unquaratine option. I also notice there are many more things listed in the local quaratntine than in the cloud version. I wonder why that is. SO far i've been fortunate to only have to deal with a few machines here and there. I symothise with those of you having to deal with hundreds or thousands.  
@, it's true there are several people posting to complain. I wasn't trying to do a don't you know who I am post. It's more of a we've been working on this since 4 pm yesterday and an official email wasn't sent out til earlier today. The reason that I mentioned I work for an MSP is because the current steps to take to get programs up and running are better designed on a small scale basis. The only quarantine restore method that seems to be working is getting the md5 hashes because files are being quarantined and not showing up in the web portal.

My current method is following the steps they have in the update today
1. Reverify
2. Rescan
3. Review quarantine on machines - means changing permissions for the workstations since it's currently locked down.
4. Check the quarantine
Grab the MD5 hashes for all the files
5. Manually add each program page at a time because it errors out when you try to do an entire site in one go.
Test and see if it's working yet and repeat the process.
I am personally disheartened by the Do You Know Who I Am attitude.
 
Shortly after some of our custom written programs started to be flagged as bad, Webroot came out and stated it was their fault. This enabled me to not chase ghosts and get our workstations cleaned using a combination of the methods described in this forum.
 
Unfortunately, most of the posts were not helpful; rather whining, complaining, demanding action. This only cluttered the minority responses that contained legitimate content. I understand many of you are upset or frustrated, but please be mindful of all of our time. Sifting through your rants to get to the actual info has been a waste of "my" time.
 
I appreciate the response to date from the Webroot employees and volunteers. Obviously, this is a difficult time, yet they have remained professional throughout. I have no plans of leaving Webroot over this incident.
 
My company is private, but we have over 700 devices with 12 sites. Even so, I don't consider myself any more important than Webroots other paying customers.
 
 
Userlevel 2
Badge +4
We got that e-mail 2.5 hours ago.
It's the same information you posted on the "Webroot False Positive" thread 4 hours ago.
 
When do we get new information?
 
 
As an MSP this has been overly frustrating. I have several clients who have been affected by this. Their production has halted due to WebRoot flagging their main applications. Trying to go through all of the machines and figuring out how to get them up and running has been an ordeal and lost multiple clients an entire day of production. I've only verified a single company has been able to get back up and running of the ones that got hammered by this.
Userlevel 7
Badge +48
For those that have not seen this email yet from Mike Malloy, Executive VP Product & Strategy, I wanted to share this with you. We sent this out to all MSP registered admins earlier today. 
 
 


 
Yesterday morning at 11:52 am MT, some good applications were mistakenly categorized as malware. This has created many false positives across the affected systems and has resulted in those applications being quarantined and unable to function. We recognize that we have not met the expectations of our customers, and are committed to resolving this complex issue as quickly as possible.
 
 Webroot is making progress on a resolution, and our entire organization is dedicated to addressing this issue.  We will update you with latest information on our Community and Blog.  In the meantime,
  • Affected customers should not uninstall the product or delete quarantine, as this will make quarantined files unrecoverable.
  • We have corrected the false positives in our backend systems, and we are working on an automated fix to reverse the false positives on endpoints. 
  • Customers should ensure that endpoints are on and connected to the Internet to receive a resolution.  Once files have been removed from quarantine, some endpoints may require rebooting.
Those who wish to address the issue manually should follow the instructions posted on Webroot Support.   We are conducting a thorough technical review to ensure we have a complete understanding of the root cause.  Once our analysis is complete, your Webroot account representatives will discuss the findings in greater detail with you. We apologize for the pain this has caused you and your customers.  Webroot appreciates your business, and our entire team is dedicated to being your most trusted partner.  We did not live up to that in this situation, but we are taking the actions to earn your trust going forward. Mike MalloyExecutive VP Product & Strategy
Userlevel 2
 
any update as of yet? 
 
Userlevel 5
Badge +24
@ wrote:
Boy there were a lot of failures and weaknesses shown with this event.  Authenticode catalogs not honored, C&C overwhelmed, no notification to users and QC inadeqacies.  I hope webroot takes this wakeup call VERY seriously and makes these shortcomings their top priority.  No more features until this is addressed.
Webroot ignoring customer-created whitelists would be a weakness too.
 
We have multiple whitelist exceptions for all files in specific file paths.
 
Webroot steamrolled right over those exceptions this time around, ignoring them and marking files infected anyway.
 
Since when would this be okay behavior by an update?  I've been told by others they experienced this as well.

Reply