Solved

W32.Trojan.Gen. False Positive Fix - April 24



Show first post

289 replies

We are also an MSP and between all of the clients that this has affected it has cost 10's of thousands of dollars in downtime.  Yet again, the anti-virus becomes the virus.  Everyone that uses Webroot, or any other AV is putting an unbelieveable amount of faith in your company to keep their company safe from not only viruses, but from instances like this.
 
We love Webroot, and this should go without saying, but please beef up your testing environment and your testing processes to ensure that this doesn't happen again.
Userlevel 1
Well, it seems we were luckier than most MSPs: We had two servers and two workstations that were mildly affected before we uninstalled Webroot from every endpoint we manage.
 
We don't know what to do now, though. Is it safe to reinstall? Is the problem going to resurface, or has it been fully resolved?
Userlevel 7
Badge +33
Hey @
 
The rule, upon discovery has been removed and they are working on a more permanent fix to repair some of the damage.
 
So yes, it's safe to put the agent onto the systems.
 
I also got away pretty lucky with only about 9 systems affected out of over 5000+ endpoints I manage. 
 
John
Easy to see how a product that can protect well can just as easily shut us down.  Still fighting endpoints removing our main business software as well as our CAD design software.  Definitely making a small IT department very unpopular for decisions that were made as a protection.
 
 
Userlevel 5
Hi everyone,
Our team (Webroot development) has been working thru the night on a safe process for moving affected files out of quarantine. We needed to insure it would not create further issues. We will provide a more detailed message with current status in a little while. This will be followed by a report that will be something you can use in your discussions with your users and/ or clients. I speak for Webroot when I say we are very sorry for the aggravation this has caused you. Once things are settled down a bit, I would be happy to speak with each of you. We can set that up with your rep. More info in a bit.
Mike Malloy
EVP Products
Userlevel 1
@, are you saying the problem is not fixed? I'm confused.
@ You really need to be posting these updates on the top thread, not 130 pages deep.  You guys have utterly failed at customer communication during this. 
Userlevel 5
Hi, we will go thru all the progress the team made overnight and assess whether all files have been cleared. If your files have been restored from quarantine then you're set. See if the apps perform as intended. If so you're done. More news in a bit.
mike
@ Thanks for your reply, however, I need to know what "a little while" means. My day is about to continue from yesterday's nightmare. Since we can't restore for quarantine, we are having to go through each computer and reinstall software.
 
Two questions: 
1. Is the issue resolved?
2. ETA till we get more details on the process of moving files out of quarantine?
How can we tell if our customer files have been restored?
Hopefully not misleading by my comments, we are not having any new endpoint issues that we are aware of.  Unfortunately the hardest hit areas of our company were our Engineering and Sales Order areas which pretty much have those departments shut down.  The previous instructions for remediation of the problem did not work.  We were able to remove Webroot from a few machines, reinstall the client software and they are working to at least get some things done, but not a good feeling of having machines unprotected.
 
We are hoping the resolution in the previous message is coming quickly and does work.
We are a small MSP.  Once we heard of this issue we created a policy called "WR Screwup" that disabled realtime protection etc. Forced all endpoints to refresh configuration.  Now its 9:06EST no issues to report, reset policies back to normal and forced a refresh configuration re-scanned all endpoints with NO issues.
 
I don't know if we got lucky or what but out of 100+ endpoints 1 was affected.  False positives on Line of business software they've been using for years.  I created an exception for the files, un-quarantined the files form the GSM console, had the client "refresh configuration" and its back to running like normal.
 
We are watching or WR managed systems like a hawk.
Userlevel 2
@ - The largest problem here is that it took 12 hours to get a response from someone other than a forum moderator.  We still have not seen any communication from our Customer Engagement teams or any management.
 
MikeM,
 
While I appreacitate that you guys are working on getting this issue resolved, the communication from Webroot leaves a lot to be desired.  Also as a MSP with over 5600 active licenses, your proposed resolution of manually releasing files from quarantine is a no go. 
For the future, please learn to be upfront and keep your partners up to date.
I was lucky as well. Out of 150+ computers on 2 sites, it only effected 6 computers and 1 server. It could have been much much worse. I feel the pain for the guys/gals that have 100's of computers that they have to fix. All my restore requests were processed sometime overnight and this morning, things seem to be ok. 
 
You put your trust in a cloud based solution and sometimes, it can cripple you. It really amazes me that this happened at all. Might as well be some form of ransomware that takes over your data and locks you out.
https:///webroot/board/message?board.id=ent2&message.id=1266#M1266
 
^^^This.  I saw the report last year.
 
The fact this was left unaddressed is not acceptable.  I'd like webroot to explain their reason for not fixing this.  
 
Overall I have loved working with webroot and fortunately we had no endpoints effected by this problem yesterday, but this may be a dealbreaker.
Userlevel 1
I have never loved working with them. From taking 6 months to solve an issue with terminal servers, to this.
Userlevel 1
We had to uninstall webroot from 4 more servers this morning because whitelists were being ignored and preventing some .exe's from running. NOT FIXED.
This is a PR nightmare, the lack of communication is upsurd. My technicians, project managers, and developers have been up all night on this and they still have not slept. We are an MSP and I am the people side of our company, when I recieved the call from our techs yesterday evening they said we are on it and we will send you an email when we know more. I got an email around midnight tell us what had happened. When I started getting calls from directors and owners this morning asking if something had happened we were very transparent with our clients. The situation was company wide but we have the best techs that were able to resolve the issues overnight for most places. We did replace some hardware however $$$$$. We will be filing for compensation for this. From the business side this is unacceptable, this cannot happen I called our owner this morning. We have been very happy uptil now with WR, but this most likely will affect our bottomline and that cannot be remeded with we are sorry. We are going to need more!  
I've worked with several.  Overall I have had less problems with webroot than others and fewer infections.  But this... this is a crippling flaw and has gone unaddressed, it seems, for almost a year after being discovered.
Userlevel 1
Dear @,
 
I think it's time to put away the "Web Threat Shield Update" link and put up "HOW WE SCREWED UP Update" link instead. Over time, perhaps it could be renamed to "False Positive 4/24 issue update"
 
Sadly, your posting added no value, as there are no actionable comments you made. 
 
I'm not sure if I feel any better that not only did your company fail to alerts the MSPs, it also failed to alert the distributors, so they didn't have a chance to communicate the problem to us.
I have to assume that Webroot has a MAJOR Q/C problem.  If after 13min an update like this can cause the kind of damage it has accross my 5600 seats it COULD NOT HAVE BEEN properly Q/Ced.   WebRoot please before you release another update on your "partners" release it on your internal systems.  Also your communication has been horrible.  
Userlevel 2
I think I speak for all MSPs on here wanting more communication from Webroot. I am pretty sure you could easily setup a spam list (mailing list) we can all sign up to for critical issue alerts.
 
 If you don't know how to set one of those up I bet I can find a few IT guys on here that can gladly help with this.
 
We know mistakes happen and most of us although are very pissed off still remain loyal because over all its a good product. But the lack of communication is unbelievable!
 
I hope that you are coming with a solution PDQ.
While I agree that comms is super important and lacking the bigger issue is that this may have been avoided.  This assumes the problem was as noted in this post:
 
https://community.webroot.com/webroot/board/message?board.id=ent2&message.id=1266#M1266
 
It seems likely, but even if not I can't understand why webroot hasn't addressed authenticode.
Userlevel 1
Your steps are probably right, providing things are working as they should.
Of all my client sites, only one was affected severely, but it was (still is) a nightmare.
 
I was sent these instructions severals times last night, and followed them to restore about 435 quarantined .exe files.   I have a manufacturing facility's entire engineering department shut down today, and these steps aren't helping much.  It has worked for some files, but others are still logged as "Not received" inthe logs.
Luckily, I have a great contact onsite, and he is working at manually finding copies of the .exe files and pasting them into place and so on.    A long and tedious process.
 
Webroot was sold to me as a product that could reverse such issues with a few clicks - nice how I wasn't told that they meant a few clicks per affected machine (or that this might not even work)!
 
 

Reply