Solved

W32.Trojan.Gen. False Positive Fix - April 24



Show first post

289 replies

Userlevel 7
Badge +48
UPDATE: April 26, 2017
 
In addition to the manual fix issued Monday, April 24, we have now issued a standalone repair utility that provides a streamlined fix for business customers. It will release and restore quarantined applications to working order on the impacted endpoints. 
 
For access to the repair utility, customers should open a support ticket, or reply to your existing support ticket related to this issue.  Please include your phone number within the support ticket.
 
Our sincerest thanks to the MSP beta customers who worked with us to test and validate this repair. We appreciate the support of our customers and thank you for your patience.
Userlevel 7
Badge +48
It is not our intention to be deceptive. We want to work with customers still being affected by this issue on an individual level to make sure that the issue is completely resolved.
 
If those customers and partners can notify us via a support ticket, that is the fastest way for us to identify and assist them. We are happy to submit a ticket on your behalf if that would be helpful. 
Userlevel 7
Badge +48
@ Just talked with support and we're seeing an average response time of one hour. 
Userlevel 2
When this is over Webroot better be buying!
Userlevel 5
@ Here's the document that customers can share with their management team. 
Userlevel 1
Badge +4
Shane C. and Sarah M. reached out and were both great and answered lots of my questions. Thank you!!!
Userlevel 1
Not trying to be rude or inconsiderate, but isn't 13 minutes a long time? Your monitoring must have seen a huge spike in the first few minutes as I am still receiving emails that have been from an hour ago. I would definitely look more into safeguards. Feeling very frustrated here.
Userlevel 7
Badge +48
For those that have not seen this email yet from Mike Malloy, Executive VP Product & Strategy, I wanted to share this with you. We sent this out earlier today.
 


 
 
As a reminder, the repair utility to address the false positive issue that arose on Monday, April 24, is available. The utility will release and restore quarantined applications to working order on the affected endpoints.
 
Please note, the utility was built to address only this specific false positive issue. It will be deactivated in the future.
 
If applications are operating normally on your systems, you do not need to implement the utility.
 
To obtain the repair utility, open a support ticket, or reply to your existing support ticket related to this issue. Please include your phone number in the ticket.
 
I want to thank each of our customers and partners for their patience during this time, and we are committed to earning your trust going forward. 
 
Yours sincerely,
 
Mike Malloy
Executive VP of Product & Strategy
Userlevel 2
Is there a way to restore the files from the command prompt on the users computer? It would be awesome if we could quickly write a script in our RMM and push it down tot he clients to restore the files.
Thanks.
Userlevel 1
Yeah, you may be able to restore from quarantine in the cloud, otherwise you need to change the policy of the agent(s) to something like unmanaged (that's the only one we have that allows for client-side changes via captcha).  Cloud process:
  1. Click on the Site that "Needs Attention" (never understood why it still needs attention if the files have been quarantined).
  2. Click on the left "x Endpoints need attention" hyperlink to the endpoint.
  3. Click on the endpoint in the popup window.
  4. Select the files you need restored from quarantine.
  5. Click Restore from Quarantine.
That's a lot of steps to fix this issue on so many computers it has affected though.
Userlevel 1
The thing is too, we've been submitting suggestions and recommendations, and complaints, and questions about all sorts of parts of the Webroot management portal, which doesn't seem properly built for MSPs.  Lack of proper filtering in the "Reports" views, inability to do things that you really need to do in a central portal.  And really, over the years none of this has improved.  They're also lacking in license management/integration with major PSAs.  So Windows 10 Anniversary Update goes out... almost every computer now shows up twice.  And now the same thing for Creator's update.  And other random changes on a system, can't figure out what.  And an inability to enable a password-locked override (have to move entirely to Unmanaged profile, which does what else, loses all overrides?  Not sure) policy to be able to manually override something as needed on an endpoint.
 
Oh, and then many computers and some servers, if they shutdown improperly (at least that's the only reason we could figure out) sometimes when starting up they blue screen and won't recover due to a failed/corrupted Webroot system file.  It can't be remotely fixed on workstations unless you have vPro working.  Never saw this get fixed either and we've still seen some recent cases of this.
 
So it's not just this singular mess-up, it's a pile-up of a bunch of things in our eyes that make it a lot less usable than it should be... this just finalized it for us.  And yes, we have to apologize to customers too.  It's time to go elsewhere.
Userlevel 2
It's times like this where you wish they had listened to MSPs and understood what real reporting is. 
 
You can't even run a report that says "This is the list of files that were quarantined across your client base in the last 24 hours."
 
And the webroot alerting system is nice enough to disable alerts per site once it reaches a certain "threshold" and just tell you to log onto the console for the site...
Please let us know why facebook is being blocked. I do not care at all if you fix that specific problem today but we need reassurance that this is not a result of a breach with your network that is not under control. I have no problem being patient while your team works late tonight getting everything fixed but your customers deserve to know if there is further risk associated with todays events.
We suspended several sites to give us time to mitigate this issue, then after unsuspending, they show as expiring.  Do we need to just wait for the system to recover because of all the traffic occuring now?  The sites have been unsuspended for hours.
 
ThePhil
Level 3 Senior Tech
Computer Troubleshooters
Class Action Suit?  This is a case for legal action.
 
We have spent most of the night working on fixing the most important systems in our client environments.  We have over 200 sites, spread across 100 miles radius.  This event has damaged our Labtech and ScreenConnect server, we had to get this to work first.  Obviously the solution proposed by WR support is not going to work for us: to manually restore and intervene on each workstation?  Without our RMM and remote tool?  After spending a while on the phone with support we were told to simply re-install our LabTech/ScreenConnect server.  Up to now the cost is very high, all our techs stayed most of the night working overtime, some clients are upset and two of them are talking about compensation for the trouble.  One of our clients is a manufacturing plant and they were stopped for many hours, this client cost per hour is 25,000$  We are not able to use recovery because most of the backup server cores are affected also, some of the servers are not yet up and we look like fools.
 
Our legal advisor is discussing the possibility to do a class action against WebRoot to recover part of the cost we all had during this event, we wonder if any of you would like compensation from WR, or to take action against them for the cost incured.  As far as we are concerned the cost up to now is over 10,000$ and there is no way we can recover any of this money, unless WR would be free for a few years, but then even for free would you like to use a product that can damage all your systems within a few minutes?  Following a human error, where only ONE person can decide about what happens to all our systems? 
 
We are a serious shop and follow ITIL guidelines: what standards of the industry does WebRoot follow? 
 
As MSP we expected faster, better results to resolve this issue, WR was nice and offered support, but not a resolution.  This morning the phones start to ring, and we have nothing to say but we are sorry. 
 
Sorry will not pay back the loss, and definitely not make our clients systems work... those interested in a class action suit should post here, so we know if there is interest.
 
MikeM,
 
While I appreacitate that you guys are working on getting this issue resolved, the communication from Webroot leaves a lot to be desired.  Also as a MSP with over 5600 active licenses, your proposed resolution of manually releasing files from quarantine is a no go. 
For the future, please learn to be upfront and keep your partners up to date.
https:///webroot/board/message?board.id=ent2&message.id=1266#M1266
 
^^^This.  I saw the report last year.
 
The fact this was left unaddressed is not acceptable.  I'd like webroot to explain their reason for not fixing this.  
 
Overall I have loved working with webroot and fortunately we had no endpoints effected by this problem yesterday, but this may be a dealbreaker.
Well, since we cannot reply to the most recent update . . . 
 
"
  • We have rolled back the false positives. Once the fix is deployed, the agent should pick up the re-determinations and perform as normal.
"
 
What does that mean? Has the fix not been deployed? Are we waiting for a fix to be deployed? Do we need to deploy a manual fix?
Userlevel 2
I find it frustrating that you put out a new thread, but don't allow anyone to reply, when there are still issues that partners are facing.  This feels like you are trying to silence people and point people to a new thread stating everything is OK.  
  
Can you please comment when the fix is going out and what partners should be doing in the meantime to remediate their situations.  Some of us don't have the ability to refresh configuration if we've hidden the tray icon.  
  
 
Userlevel 2
Thank you for responding to point 1
 
Can you comment on point 2 for me and all of the users/partners that are affected.
 
Has the fix been pushed out, if not, when will the global fix be pushed out?  
 
I would ask other important questions but at the risk of questions being selectively answered I'll limit it to one at a time.
A few notes for people who are working to resolve the issue on a local level.
 
We are still noting that attempts at restoring quarantined files from the cloud are not working.  We are using the 'Unmanaged' profile to access local Quarantined files.
 
When applying the 'Unmanaged' profile, you may use WRSA.exe -poll to immediately enforce the change from the local machine.  For our cloud instance, that is working quickly.  I suspect that the cloud instances for some of the larger MSPs here are under greater load (at the risk of understating the issue)
 
If for some reason, you cannot access Quarantined files, sometimes they will be in C:Quarantine as the restore command you issued sometime earlier this week was unable to restore to the prior location.  You MAY have the option to use 'previous versions' of a folder (ie. Windows Shadow Storage) to pull your files out of the nether.  
 
I thought we were all caught up last night and found that a fair number of customers were affected and not flagged, and that I did not receive email alerting for all endpoints with issues - I would recommend anyone with multiple organizations to run a report showing all detections in the last 24 hours in order to make sure your bases are covered.
 
I hope everyone has enough coffee to get through the day.
 
 
" We have rolled back the false positives. Once the fix is deployed, the agent should pick up the re-determinations and perform as normal."
 
Has the fix been deployed yet? It is not clear.  What about files still in quarantine? Do we have restore from Quarantine (again)? Since we already did, yesterday, and it clearly does not work.
 
Will our files automatically be restored?
 
Please provide us with more details on this fix and how and when we will get it.
 
Thanks
 
Userlevel 7
Badge +35
Please find the letter that our MSP partners may share with their customers who were affected here:
 
https:///webroot/attachments/webroot/ent2/1451/1/WebrootIncidentResponseLetter.pdf
Userlevel 2
@ Do you mean switching the policies to unmanaged or waiting on the command as it sits at "Not yet received"?
 
If it's for switching policies, I have not encountered any issue. Here is what I did in switching policies:
 
1.) Go to Webroot Console
2.) Endpoint Protect > Find the end point > Apply policy to endpoints > Unmanaged
2.) Log into the workstation, right click on the Webroot agent, hit Refresh Configuration
3.) Once policy is confirmed, Righ click Webroot > Open > PC Security (Cog next to it) > Quarantine > Select all that Apply > Restore
4.) This may have a while to release.
5.) Once complete, you can put the end point back into managed mode. (You can also run a verification but by that point, I didn't need to)
 
As for waiting on the commands, it took several hours for commands to be pushed as there is a huge backlog. Last night it took over 8 hours for the backlog to catch up. You can attempt to poll via the command line but I had very little success in this. 
 For 32bit Operating Systems, type:       "C:Program FilesWebrootWRSA.exe" -poll For 64bit Operating Systems, type:       "C:Program Files (x86)WebrootWRSA.exe" -poll  You have to make sure that the endpoint is online or it won't work! Had this happen to several today where the endpoint showed online but the workstation wasn't. I've only encountered 1 issue through all the endpoints I went through ( 200 + ). If you need anything else, feel free to PM me. 🙂

Reply