Solved

W32.Trojan.Gen. False Positive Fix - April 24



Show first post

289 replies

I was lucky as well. Out of 150+ computers on 2 sites, it only effected 6 computers and 1 server. It could have been much much worse. I feel the pain for the guys/gals that have 100's of computers that they have to fix. All my restore requests were processed sometime overnight and this morning, things seem to be ok. 
 
You put your trust in a cloud based solution and sometimes, it can cripple you. It really amazes me that this happened at all. Might as well be some form of ransomware that takes over your data and locks you out.
MikeM,
 
While I appreacitate that you guys are working on getting this issue resolved, the communication from Webroot leaves a lot to be desired.  Also as a MSP with over 5600 active licenses, your proposed resolution of manually releasing files from quarantine is a no go. 
For the future, please learn to be upfront and keep your partners up to date.
Userlevel 2
@ - The largest problem here is that it took 12 hours to get a response from someone other than a forum moderator.  We still have not seen any communication from our Customer Engagement teams or any management.
 
We are a small MSP.  Once we heard of this issue we created a policy called "WR Screwup" that disabled realtime protection etc. Forced all endpoints to refresh configuration.  Now its 9:06EST no issues to report, reset policies back to normal and forced a refresh configuration re-scanned all endpoints with NO issues.
 
I don't know if we got lucky or what but out of 100+ endpoints 1 was affected.  False positives on Line of business software they've been using for years.  I created an exception for the files, un-quarantined the files form the GSM console, had the client "refresh configuration" and its back to running like normal.
 
We are watching or WR managed systems like a hawk.
Hopefully not misleading by my comments, we are not having any new endpoint issues that we are aware of.  Unfortunately the hardest hit areas of our company were our Engineering and Sales Order areas which pretty much have those departments shut down.  The previous instructions for remediation of the problem did not work.  We were able to remove Webroot from a few machines, reinstall the client software and they are working to at least get some things done, but not a good feeling of having machines unprotected.
 
We are hoping the resolution in the previous message is coming quickly and does work.
How can we tell if our customer files have been restored?
@ Thanks for your reply, however, I need to know what "a little while" means. My day is about to continue from yesterday's nightmare. Since we can't restore for quarantine, we are having to go through each computer and reinstall software.
 
Two questions: 
1. Is the issue resolved?
2. ETA till we get more details on the process of moving files out of quarantine?
Userlevel 5
Hi, we will go thru all the progress the team made overnight and assess whether all files have been cleared. If your files have been restored from quarantine then you're set. See if the apps perform as intended. If so you're done. More news in a bit.
mike
@ You really need to be posting these updates on the top thread, not 130 pages deep.  You guys have utterly failed at customer communication during this. 
@, are you saying the problem is not fixed? I'm confused.
Userlevel 5
Hi everyone,
Our team (Webroot development) has been working thru the night on a safe process for moving affected files out of quarantine. We needed to insure it would not create further issues. We will provide a more detailed message with current status in a little while. This will be followed by a report that will be something you can use in your discussions with your users and/ or clients. I speak for Webroot when I say we are very sorry for the aggravation this has caused you. Once things are settled down a bit, I would be happy to speak with each of you. We can set that up with your rep. More info in a bit.
Mike Malloy
EVP Products
Easy to see how a product that can protect well can just as easily shut us down.  Still fighting endpoints removing our main business software as well as our CAD design software.  Definitely making a small IT department very unpopular for decisions that were made as a protection.
 
 
Userlevel 7
Badge +30
Hey @
 
The rule, upon discovery has been removed and they are working on a more permanent fix to repair some of the damage.
 
So yes, it's safe to put the agent onto the systems.
 
I also got away pretty lucky with only about 9 systems affected out of over 5000+ endpoints I manage. 
 
John
Well, it seems we were luckier than most MSPs: We had two servers and two workstations that were mildly affected before we uninstalled Webroot from every endpoint we manage.
 
We don't know what to do now, though. Is it safe to reinstall? Is the problem going to resurface, or has it been fully resolved?
We are also an MSP and between all of the clients that this has affected it has cost 10's of thousands of dollars in downtime.  Yet again, the anti-virus becomes the virus.  Everyone that uses Webroot, or any other AV is putting an unbelieveable amount of faith in your company to keep their company safe from not only viruses, but from instances like this.
 
We love Webroot, and this should go without saying, but please beef up your testing environment and your testing processes to ensure that this doesn't happen again.
Userlevel 2
After spending a good part of my evening and this morning going through servers and vital PCs, many of our endpoints are reporting that the quarantine released the files. This took over 12 hours to complete and we aren't out of the woods just yet. Now the part to face our clients who are going to have a lot of questions and rightfully so, anger. I understand that mistakes happen. I don't believe anyone on this forum can say that they haven't at least 1 big time mess up. It's just the lack of communication that hurts the most. I'm our primary POC for Webroot and I feel bad for our Account Managers because I was ringing their phones off the hook demanding an answer like so many. At least he was very nice about it and kept me in the loop but if I hadn't, I feel I would have been left to my own devices. 😕
Any further updates on this? We have thousands of endpoints running Webroot across hundreds of client sites. This total lack of transparency and communication is appalling.
We called support about the sites showing "expired" or "expiring". The guy on the phone said he can usually push through the change to make the site "protected", but it wasn't responding for him. He "sent the request up the chain" and asked them to go into our site and re-enabled all 82 of them that are showing expiring. 
 
That was last night around 9:00PM Central. This morning, they're all still showing expiring. :(
@ wrote:
@ @ @ @ Please contac customer support at 1-866-254-8400 so that they can troubleshoot this further.  
 
@ Still no updates or official word for MSPs?
Userlevel 1
We're still seeing files moved to quarantine as of this morning at 7:34 EDT.
 
WR just shutdown Ben and Jerry's deliverys for New England.
 
So much for the problem being resolved yesterday.
@Keep me posted on this.  This is the 2nd time this year that WebRoot has screwed the pooch.  Also an MSP here, and I'm spending this morning doing multiple songs and dances for my clients to explain to them how we're on top of it and they'll never see webroot again.
@ Count us in. The complete lack of comunication has been a desaster that has only compounded the feeling that those of us with hundreds of frustrated clients are being ignored.
 
Just the cost for our own staff to deal with issues steming from this is going to quickly exceed 5k today. I hate to think about the per client cost... 
Class Action Suit?  This is a case for legal action.
 
We have spent most of the night working on fixing the most important systems in our client environments.  We have over 200 sites, spread across 100 miles radius.  This event has damaged our Labtech and ScreenConnect server, we had to get this to work first.  Obviously the solution proposed by WR support is not going to work for us: to manually restore and intervene on each workstation?  Without our RMM and remote tool?  After spending a while on the phone with support we were told to simply re-install our LabTech/ScreenConnect server.  Up to now the cost is very high, all our techs stayed most of the night working overtime, some clients are upset and two of them are talking about compensation for the trouble.  One of our clients is a manufacturing plant and they were stopped for many hours, this client cost per hour is 25,000$  We are not able to use recovery because most of the backup server cores are affected also, some of the servers are not yet up and we look like fools.
 
Our legal advisor is discussing the possibility to do a class action against WebRoot to recover part of the cost we all had during this event, we wonder if any of you would like compensation from WR, or to take action against them for the cost incured.  As far as we are concerned the cost up to now is over 10,000$ and there is no way we can recover any of this money, unless WR would be free for a few years, but then even for free would you like to use a product that can damage all your systems within a few minutes?  Following a human error, where only ONE person can decide about what happens to all our systems? 
 
We are a serious shop and follow ITIL guidelines: what standards of the industry does WebRoot follow? 
 
As MSP we expected faster, better results to resolve this issue, WR was nice and offered support, but not a resolution.  This morning the phones start to ring, and we have nothing to say but we are sorry. 
 
Sorry will not pay back the loss, and definitely not make our clients systems work... those interested in a class action suit should post here, so we know if there is interest.
 
Webroot any update on this, I have about 30 minutes before a bunch of pissed off customers start in on my support staff.
Still not working... created override and restore from quarantine still not working as of 11pm PST
Any update from webroot?

Reply