Solved

W32.Trojan.Gen. False Positive Fix - April 24


Userlevel 7
Badge +48
Update April 28, 11:45 a.m. MDT: 
 
Please click here to see the most recent update.
 
 
UPDATE 4/28/17 11:45 a.m. MNT: We have 0 calls in queue on our phone line, and are working through about 80 tickets related to the False Positive repair utility. A good portion of those are simply awaiting customer verification.
 
Please note, the utility was built to address only this specific false positive issue. It will be deactivated in the future. 
 
If applications are operating normally on your systems, you do not need to implement the utility. 
 
If you haven’t yet submitted a support ticket and you need the repair utility, please do so here. Include your phone number as well with the support ticket.
 
Thank you.
 
icon

Best answer by freydrew 26 April 2017, 18:25

UPDATE: April 26, 2017

 

In addition to the manual fix issued Monday, April 24, we have now issued a standalone repair utility that provides a streamlined fix for business customers. It will release and restore quarantined applications to working order on the impacted endpoints. 

 

For access to the repair utility, customers should open a support ticket, or reply to your existing support ticket related to this issue.  Please include your phone number within the support ticket.

 

Our sincerest thanks to the MSP beta customers who worked with us to test and validate this repair. We appreciate the support of our customers and thank you for your patience.
View original

289 replies

Userlevel 7
Hi @, and welcome to our Community!
 
 
The actions we have taken include:
  • We immediately repaired and strengthened our safeguards related to the false positive on the day it occurred.  In the days and weeks following, we introduced a number of new safeguards – both technical and procedural – to reduce our exposure to similar incidents.
  • We scaled up our infrastructure to ensure our console performs well and supports the high volume of agent commands that are likely during any service issue.
  • We’ve improved our communication around product capabilities, updates and issues.  This includes the introduction of a series of certification programs to scale our information sharing on best practices, as it became clear that customers who had greater familiarity with the best practices in using our products were able to resolve issues in their environments and return to normal operations faster.  (link to partner certification: https://www.webroot.com/us/en/about/press-room/releases/webroot-launches-certification-program)
  • Finally, we are increasing the frequency of early communication across all our channels—email, social media, support, and community—so that when issues arise, the likely impact and status of remediation are shared out as quickly as possible.  
 
If there are specific questions we can answer for you, we would happily jump on a phone call with you.
I came across this thread while doing some due dilligence before I recommend adopting Webroot as the preferred security product for the MSP that I am Service Deliver Manager of. 
 
You will understand I'm interested to hear more about the improvements made in quality control and incident response to avoid the headaches that other MSPs have discussed in this thread. 
 
Can anyone direct me to a summary of changes implemented in the wake of lessons learned from this
incident? 
 
Thanks. 
Thanks.
Userlevel 5
Hi dsm55 and others
We are sending an email and posting a letter from our CEO, Dick Williams, which outlines some of the many steps we have taken already and are actively working on to 1) prevent similar issues; 2) communicate more rapidly and with better coverage; and 3) improve our systems so that you can take remediation steps yourself with better information. That note and others in the weeks ahead will hopefully provide you the assurance you need to depend on Webroot as a solid partner. We know this event was a big one and have neither dismissed it nor ignored its many lessons. Thanks for your note.
Mike
This event certainly uncovered some big issues.
 
Now that it's behind us I would like to know what your plans are for making sure the trifecta of bad (detection, backlog, no kill switch) does not happen again.  
 
I have no doubt you are taking this very seriously.  Just looking for more information.
 
Thanks.
 
Userlevel 7
Badge +48
For those that have not seen this email yet from Mike Malloy, Executive VP Product & Strategy, I wanted to share this with you. We sent this out earlier today.
 


 
 
As a reminder, the repair utility to address the false positive issue that arose on Monday, April 24, is available. The utility will release and restore quarantined applications to working order on the affected endpoints.
 
Please note, the utility was built to address only this specific false positive issue. It will be deactivated in the future.
 
If applications are operating normally on your systems, you do not need to implement the utility.
 
To obtain the repair utility, open a support ticket, or reply to your existing support ticket related to this issue. Please include your phone number in the ticket.
 
I want to thank each of our customers and partners for their patience during this time, and we are committed to earning your trust going forward. 
 
Yours sincerely,
 
Mike Malloy
Executive VP of Product & Strategy
Userlevel 7
Badge +48
UPDATE 4/28/17 11:44 a.m. MNT: We have 0 calls in queue on our phone line, and are working through about 80 tickets related to the False Positive repair utility. A good portion of those are simply awaiting customer verification.
 
Please note, the utility was built to address only this specific false positive issue. It will be deactivated in the future. 
 
If applications are operating normally on your systems, you do not need to implement the utility. 
 
If you haven’t yet submitted a support ticket and you need the repair utility, please do so here. Include your phone number as well with the support ticket.
 
Thank you.
Userlevel 7
Badge +48
For those that have not seen this email yet from Mike Malloy, Executive VP Product & Strategy, I wanted to share this with you. We sent this out earlier today.
 


 
We want to remind you that we have created a repair utility to address a false positive issue that arose on Monday.  
 
On April 24 at 11:52 am MT, some good applications were mistakenly categorized by Webroot as malware. This created false positives across the affected systems and resulted in those applications being quarantined and unable to function. 
 
Our repair utility will release and restore quarantined applications to working order on the affected endpoints.  
 
To obtain the repair utility, please open a support ticket, or reply to your existing support ticket related to this issue. Please include your phone number in the ticket.  
 
We appreciate the support of our customers and partners, and thank you for your patience.
 
Yours sincerely,
 
Mike Malloy
Executive VP of Product & Strategy
Userlevel 7
Badge +48
UPDATE 4/27/17 2:46 p.m. MNT: We have 0 calls in queue on our phone line, and are working through about 130 tickets related to the False Positive repair utility. A good portion of those are simply awaiting customer verification.
 
If you haven’t yet submitted a support ticket and you need the repair utility, please do so here. Include your phone number as well with the support ticket.
 
Our sincerest thanks to the MSP beta customers who worked with us to further test and validate this repair. We truly appreciate the support of our customers and thank you for your patience.
Userlevel 1
@ thanks for the suggestion. Shortly after I posted, Shane reached out to me to assit. Kudos to the support guys that have been beaten up over the last few days and still aggressively working to make sure everything is perfect again!
Userlevel 7
Hey, @.
 
These are currently unknown issues from the false positives, so it'd be a good idea for you to reach out to our Support Team directly.
 
Business Technical Support: Call 1-866-254-8400
Open a Support Ticket
Userlevel 1
@ Glad to hear things seem to be calming down for you guys! I know it's been a difficult week and am very appreciative of the help I received from Shane, Brandon, Greg, and the other guys I've corresponded with over phone or email.
 
Still have a few concerns today. I still see agents with commands "Not yet recieved" in the console going back to 4/24 and 4/25. Any idea when this will clear up or be addressed?
 
I also have 26 of the 138 sites I have in GSM showing that that "Need attention" though I'm sure that the majority of that number do not need attention any longer. I'm sure one or two of those may be legitimate but certainly not all.
 
Is this behavior expected at this point or do I need to get back on the line with your Support Team?
 
Thanks,
Jared
Userlevel 7
Badge +48
UPDATE 4/27/17 9:21 a.m. MNT: We have 0 calls in queue on our phone line, and are working through about 100 tickets related to the False Positive repair utility. A good portion of those are simply awaiting customer verification.
 
If you haven’t yet submitted a support ticket and you need the repair utility, please do so here. Include your phone number as well with the support ticket.
 
Our sincerest thanks to the MSP beta customers who worked with us to further test and validate this repair. We truly appreciate the support of our customers and thank you for your patience.
 
Userlevel 1
I can confirm that it is still happening. Shut down another distributor client of mine. The server is set to ALL DISABLED, but it's possible a workstation did the damage. However, all of them are supposed to be set to "all disabled" as well.
Userlevel 1
Didn't opt in for beta fix:
 
Agent refused to checkin to cloud console.

-Booted workstation to safe mode
-WRSA -uninstall
-Reinstalled
 
Agent now checks in, no new false positives yet.
Userlevel 1
Badge +4
Shane C. and Sarah M. reached out and were both great and answered lots of my questions. Thank you!!!
Userlevel 7
Badge +35
@, I spoke with our support team and asked our SEs to reach out to you. Can you please let us know if your case was resolved?
Userlevel 7
Badge +48
@ One possible reason why you're experiencing this would be if the system were low on drive space when the issue happened. I'd recommend talking with support and submitting a ticket so that they can further assist you. https://www.webroot.com/us/en/about/contact-us 
 
Thanks!
Userlevel 5
Please contact support. They can address your issues with you. 
Mike
Userlevel 1
Badge +4
Hello. I ran the tool on one of the infected machines, it disabled my Webroot. How do we know when the fix is done? After 10 minutes I enabled Webroot again and it ran a scan and the file that was falsely identified was once again found as a threat, I allowed it ... but I feel like I'm still not back to normal. I even had support (and myself), whitelist that file and folder in my console. 
 
I have around 350 endpoints, it took out 3 security cameras and quoting software, and then some facebook pages. This has caused a lot of headaches and I've only had a few issues. Can't imagine if it would have taken out servers and all my workstations. I'm still not feeling very confident with Webroot now, and to make matters worse my Account Rep never replies to my e-mails. 
Userlevel 5
@ Here's the document that customers can share with their management team. 
Userlevel 2
MSP Update:
 
So far I am seeing good success with this utility. have only run into 3 workstations so far that are not working.
 
I am not getting console commands to run at this time still... hoping that gets resolved as well>??
Is anyone else having issues with files upwards of 20GB being created from this, filling up hard drive space?
Userlevel 5
Hi No, we dont want to hang on the phone with your team while you use it. They want to understand (quickly) if the person requesitng the tool is confident in his abiltiy to just run with it. You know some people ask for tools like this who dont actually know what theyre doing. So you will be given access to download it very quickly once you connect with support.
Mike
I requested the "fix" through the already-open ticket.  The response can be summarized by "We would like to schedule an appointment to call you, during which a member of the Webroot Advanced Malware Removal Team will provide remote assistance on the affected computer."  It appears that Webroot wants me to be on the phone with their support engineers for several days deploying this to all of our machines.

Reply