Please click here to see the most recent update.
UPDATE 4/28/17 11:45 a.m. MNT: We have 0 calls in queue on our phone line, and are working through about 80 tickets related to the False Positive repair utility. A good portion of those are simply awaiting customer verification.
Please note, the utility was built to address only this specific false positive issue. It will be deactivated in the future.
If applications are operating normally on your systems, you do not need to implement the utility.
If you haven’t yet submitted a support ticket and you need the repair utility, please do so here. Include your phone number as well with the support ticket.
Best answer by freydrewView original
Can you comment on point 2 for me and all of the users/partners that are affected.
Has the fix been pushed out, if not, when will the global fix be pushed out?
I would ask other important questions but at the risk of questions being selectively answered I'll limit it to one at a time.
[edited by community moderator] Regardless, all those unanswered questions are quite valid and for webroot to not have an answer after 24hrs is quite disheartening. [edited by community moderator]
Anyways, this issue has created great damaged at our clients and your fix (automatic or manual) have still left some applications in an inoperable state (forcing us to either repair or reinstall the software).
As MSPs, we ask Webroot to be more up front and communicate with your partners.
Lastly, Webroot's update about not deleting files from quarantine is quite exasperating. Do you believe having to wait for over 24hrs for a resolution is an acceptable path? We had to do whatever we could to get our clients back in business. [edited by community moderator]
We are still noting that attempts at restoring quarantined files from the cloud are not working. We are using the 'Unmanaged' profile to access local Quarantined files.
When applying the 'Unmanaged' profile, you may use WRSA.exe -poll to immediately enforce the change from the local machine. For our cloud instance, that is working quickly. I suspect that the cloud instances for some of the larger MSPs here are under greater load (at the risk of understating the issue)
If for some reason, you cannot access Quarantined files, sometimes they will be in C:Quarantine as the restore command you issued sometime earlier this week was unable to restore to the prior location. You MAY have the option to use 'previous versions' of a folder (ie. Windows Shadow Storage) to pull your files out of the nether.
I thought we were all caught up last night and found that a fair number of customers were affected and not flagged, and that I did not receive email alerting for all endpoints with issues - I would recommend anyone with multiple organizations to run a report showing all detections in the last 24 hours in order to make sure your bases are covered.
I hope everyone has enough coffee to get through the day.
Thanks for that we will try that now. On the third pot this morning but got no sleep last night. Thanks for the insight I am willing to try just about anything at this point...
Can a Webroot Employee or support please let us know an ETA, lie to me I don't care but at least give us some hope of sleep today.
I've been paying very close attention to this failure and been up most of the night trying to monitor and see if we were affected in any way. We have seemingly dodged a bullet here but that's not to say we are out of the woods by any means. We can't seem to pinpoint how we avoided the failure but only point to the fact that our scan times are set to 11pm-3am everyday as opposed to daytime defaults that Webroot uses. I can't say for certain that this helped us avoid the disaster, but we can't find any other reason as to why we got lucky here. Our guess is that because the update went out yesterday morning, and deep scans occurred after that for endpoints, perhaps that's when it flagged .exe's as false positives and quarantined them. And since we didn't scan during that period, and Webroot released the first fix during the afternoon, we missed that window.
Not sure if you guys will find this helpful, but if Webroot is issuing these updates during the day, perhaps think about changing your deeps scans for afterhours to avoid that window of updating to see if there are any issues and allows Webroot time to fix the problems. (Not that this should have happened in the first place)
Where is our official statement? Where is our comprehensive solution? Are we going to have to deal with the effects of this tomorrow as well?
Has the fix been deployed yet? It is not clear. What about files still in quarantine? Do we have restore from Quarantine (again)? Since we already did, yesterday, and it clearly does not work.
Will our files automatically be restored?
Please provide us with more details on this fix and how and when we will get it.
Has anyone found an effective fix other than wiping the machines? Removing the WRkrn.sys file (which works for a botched Webroot install) does not workin in this instance due to the mangled files from this issue.
Any help would be awesome!
We have multiple whitelist exceptions for all files in specific file paths.
Webroot steamrolled right over those exceptions this time around, ignoring them and marking files infected anyway.
Since when would this be okay behavior by an update? I've been told by others they experienced this as well.
any update as of yet?
Yesterday morning at 11:52 am MT, some good applications were mistakenly categorized as malware. This has created many false positives across the affected systems and has resulted in those applications being quarantined and unable to function. We recognize that we have not met the expectations of our customers, and are committed to resolving this complex issue as quickly as possible.
Webroot is making progress on a resolution, and our entire organization is dedicated to addressing this issue. We will update you with latest information on our Community and Blog. In the meantime,
It's the same information you posted on the "Webroot False Positive" thread 4 hours ago.
When do we get new information?
Shortly after some of our custom written programs started to be flagged as bad, Webroot came out and stated it was their fault. This enabled me to not chase ghosts and get our workstations cleaned using a combination of the methods described in this forum.
Unfortunately, most of the posts were not helpful; rather whining, complaining, demanding action. This only cluttered the minority responses that contained legitimate content. I understand many of you are upset or frustrated, but please be mindful of all of our time. Sifting through your rants to get to the actual info has been a waste of "my" time.
I appreciate the response to date from the Webroot employees and volunteers. Obviously, this is a difficult time, yet they have remained professional throughout. I have no plans of leaving Webroot over this incident.
My company is private, but we have over 700 devices with 12 sites. Even so, I don't consider myself any more important than Webroots other paying customers.
My current method is following the steps they have in the update today
3. Review quarantine on machines - means changing permissions for the workstations since it's currently locked down.
4. Check the quarantine
Grab the MD5 hashes for all the files
5. Manually add each program page at a time because it errors out when you try to do an entire site in one go.
Test and see if it's working yet and repeat the process.
As someone pointed out, it wasn't so much of a "DO YOU KNOW WHO I AM?" but more of a reaction that the initial fix wasn't a feasible option for an MSP who deals with thousands of endpoints. And then when pointed out that this would not work for an MSP who does have thousands of endpoints, it just felt that the ball was dropped and we were left in the dark. Don't get me wrong, I like Webroot's product. I've worked with them for several years and will most likely continue working with them. We can't fix the past but I would like to know how Webroot plans to correct this so this doesn't occur again. What tools can they provide to us so that we can put a stop (if possible) instead of waiting more than a day. I commend all the Webroot techs and the sales reps that I've spoken with. They've been understanding and I know my day hasn't been as bad as theirs.
Anyways, one thing I did notice is that even though the command for releasing the quarantine had executed, we still had to manually remote to the computers. And then there was one odd computer that even though we released the quarantine and never actually released the quaratine. We then discovered that even though it was in an unmanaged mode, all of the .exe were set to be "blocked". Once we unblocked these, everything released and the workstation began working normally. Hope that helps anyone who encounter that issue like we did. :)
-Switch policy to unmanged in the Dashboard
-on local device, right click and Refresh Dashboard on WR tray icon
-on local device release the quaratined files
-on dashboard run the "Re-verify all file...." Agent command
-on local device, right click and Refresh Dashboard on WR tray icon
-Switch policy to back to the actual policy in the Dashboard