Solved

W32.Trojan.Gen. False Positive Fix - April 24



Show first post

289 replies

Is it possible to restore the latest working state: applications, files, system images; and fallback to the most previous working Webroot version?
Userlevel 2
@
 
Thanks for that we will try that now. On the third pot this morning but got no sleep last night.  Thanks for the insight I am willing to try just about anything at this point...
 
Can a Webroot Employee or support please let us know an ETA, lie to me I don't care but at least give us some hope of sleep today.
 
:mansad:
Userlevel 7
Badge +35
@ we are fully focused on resolving this issue and do not want to spend time defending false rumors. Please continue posting your questions, but kindly refrain from adding rumors and false information. Your post has been edited to remove the offending comments. You may also review the Community Guidelines here.
@ I almost appreciate the "all hands-on deck" statement BUT in as much as this is making the national news, affecting business for so many companies, detrimentally affection client relationships with MSPs all over, not to mention the huge financial impact to both clients and MSPs, I would have expected Webroot to be much more transparent and responsive. As it is the response from Webroot appears to me to be extraordinarily apathetic.

Where is our official statement? Where is our comprehensive solution? Are we going to have to deal with the effects of this tomorrow as well?
 
Regards,
Mark G
Userlevel 7
Badge +48
UPDATE: We've got an update on the initial post in this thread. This update includes further messaging around addressing the issue manually. We are conducting a thorough technical review to ensure we have a complete understanding of the root cause. I wanted to make sure that all of our subscribers got the message. Please continue the discussion in this forum. Thanks.
Userlevel 1
Any word on a mass-scale fix? This is painful.
I'd really like to know if anyone has any idea on machines bluescreening after they get to the login screen in Windows 7/10 after this. Just bootlooping over and over. 
 
Has anyone found an effective fix other than wiping the machines? Removing the WRkrn.sys file (which works for a botched Webroot install) does not workin in this instance due to the mangled files from this issue.
 
Any help would be awesome!
Had one this morning but it was an machine we were in the process of deploying Webroot on and removing AVG Cloudcare. AVG had not removed yet so we manually removed AVG and then all was fine. File causing the BSOD error was netio.sys.
Userlevel 2
 
any update as of yet? 
 
Userlevel 7
Badge +48
For those that have not seen this email yet from Mike Malloy, Executive VP Product & Strategy, I wanted to share this with you. We sent this out to all MSP registered admins earlier today. 
 
 


 
Yesterday morning at 11:52 am MT, some good applications were mistakenly categorized as malware. This has created many false positives across the affected systems and has resulted in those applications being quarantined and unable to function. We recognize that we have not met the expectations of our customers, and are committed to resolving this complex issue as quickly as possible.
 
 Webroot is making progress on a resolution, and our entire organization is dedicated to addressing this issue.  We will update you with latest information on our Community and Blog.  In the meantime,
  • Affected customers should not uninstall the product or delete quarantine, as this will make quarantined files unrecoverable.
  • We have corrected the false positives in our backend systems, and we are working on an automated fix to reverse the false positives on endpoints. 
  • Customers should ensure that endpoints are on and connected to the Internet to receive a resolution.  Once files have been removed from quarantine, some endpoints may require rebooting.
Those who wish to address the issue manually should follow the instructions posted on Webroot Support.   We are conducting a thorough technical review to ensure we have a complete understanding of the root cause.  Once our analysis is complete, your Webroot account representatives will discuss the findings in greater detail with you. We apologize for the pain this has caused you and your customers.  Webroot appreciates your business, and our entire team is dedicated to being your most trusted partner.  We did not live up to that in this situation, but we are taking the actions to earn your trust going forward. Mike MalloyExecutive VP Product & Strategy
Userlevel 1
Badge +4
We got that e-mail 2.5 hours ago.
It's the same information you posted on the "Webroot False Positive" thread 4 hours ago.
 
When do we get new information?
 
 
@, it's true there are several people posting to complain. I wasn't trying to do a don't you know who I am post. It's more of a we've been working on this since 4 pm yesterday and an official email wasn't sent out til earlier today. The reason that I mentioned I work for an MSP is because the current steps to take to get programs up and running are better designed on a small scale basis. The only quarantine restore method that seems to be working is getting the md5 hashes because files are being quarantined and not showing up in the web portal.

My current method is following the steps they have in the update today
1. Reverify
2. Rescan
3. Review quarantine on machines - means changing permissions for the workstations since it's currently locked down.
4. Check the quarantine
Grab the MD5 hashes for all the files
5. Manually add each program page at a time because it errors out when you try to do an entire site in one go.
Test and see if it's working yet and repeat the process.
     I've been having the same experience where restoring quarantine from the Cloud portal does nto seem to work. I've had to use the unmanaged polciy and local unquaratine option. I also notice there are many more things listed in the local quaratntine than in the cloud version. I wonder why that is. SO far i've been fortunate to only have to deal with a few machines here and there. I symothise with those of you having to deal with hundreds or thousands.  
Userlevel 2
Things have died down on our end. This morning was a little rough but not as bad as it could have been. I spent a greater part of my evening lastnight adding exceptions and then running the quarantine release one-by-one (Beer and ice cream helped). I woke up this morning at 5 AM to check on the progress and most of the backlog had caught up. This gave me hope that the world wasn't ending. It was just going to take time.
 
As someone pointed out, it wasn't so much of a "DO YOU KNOW WHO I AM?" but more of a reaction that the initial fix wasn't a feasible option for an MSP who deals with thousands of endpoints. And then when pointed out that this would not work for an MSP who does have thousands of endpoints, it just felt that the ball was dropped and we were left in the dark. Don't get me wrong, I like Webroot's product. I've worked with them for several years and will most likely continue working with them. We can't fix the past but I would like to know how Webroot plans to correct this so this doesn't occur again. What tools can they provide to us so that we can put a stop (if possible) instead of waiting more than a day. I commend all the Webroot techs and the sales reps that I've spoken with. They've been understanding and I know my day hasn't been as bad as theirs.
 
Anyways, one thing I did notice is that even though the command for releasing the quarantine had executed, we still had to manually remote to the computers. And then there was one odd computer that even though we released the quarantine and never actually released the quaratine. We then discovered that even though it was in an unmanaged mode, all of the .exe were set to be "blocked". Once we unblocked these, everything released and the workstation began working normally. Hope that helps anyone who encounter that issue like we did. :)
 
 
Userlevel 2
@ I ran into this exact issue on several workstations today. Mostly it was catching our remote agent installer in our %netlogon% on all servers but then it was doubling and even tripling. I hadn't seen it on all of the workstations but I did find it quite odd that it wasn't showing up in the cloud console. 
Userlevel 1
 
Also does anyone know of a way to get agents to unmanaged if console command is not working????
 
This would be beyond valuable. I have had some success getting commands pushed to workstations but a few refuse to push out. Any suggestions would be appreciated.
Userlevel 7
Badge +48
@ Thanks for the heads up. 
I have called the number and a ticket was created for me. She said she would send it to escalations in Australia because the US office is closed. I didn't see the note about Sean or Lucas. Ticket number is 85056. I would like to test.

I do want to say after manually cleaning a few computers, they did get flagged again with quarantined files. The machine are XP.
Userlevel 7
Badge +48
@ Thank you! I've notified Shawn about this issue and he'll reach out. 
Userlevel 7
Badge +48
UPDATE: We continue to make progress on a resolution to our false positive issue. 
 
We created a comprehensive repair utility, and have successfully completed QA. We are currently rolling out the utility to a group of beta customers to ensure it works for our broader customer base. We expect to complete that work soon, and then will make it available incrementally to the entire customer base to ensure a successful deployment.
 
Stay here for ongoing updates.
 
Our Support team remains available to those of you who need urgent assistance, and we thank you for working with us through this challenging issue.
 
Can anyone offer suggestions for a computer that has been almost completely disabled by these false positives? I cannot boot to safe mode or safe mode with networking to performing any of the gui-based fixes. sfc /scannow via windows 10 startup troubleshooting is unsuccessful. I was able to capture the dlb.db file from the broken laptop's hard drive (per tech support's instructions) and load it on an unmanaged webroot endpoint to view all of the files that were quarantined. The list has got to be 200-300 items long and is 95% registry entries. How can I get this broken laptop working again? Support has suggested various fixes (none of which have worked) but has also stated that these fixes don't apply to the registry files that were incorrectly quarantined. Help!
Userlevel 1
@ you might be boned dude. Good luck.
I saw the letter MSPs can use, but what about regular small businesses?  I would like an official letter too that I can share with executives that are beating down my door.  
 
Thanks!
@ I would snag a Windows 10 repair disk/USB image and try to repair from boot.  What a mess, I'm sorry you have to deal with that.  
Userlevel 1
Good morning.
 
I arrived today to find some PC's that had been repaired yesterday exhibiting the same behavior today. Key notes:
 
  1. We have followed previous recommendations for MSP's
  2. Files were added to the exclusion list both with MD5 hashes as well as by folder path.
  3. Application files were flagged again but this time with a different infection.
  4. The malware group reported is win32.autoblock.1
  5. I was again able to restore from quarantine as we are now running almost every client as "Unmanaged"
  6. I ran the wsalogs.exe utility to grab full diagnostic logs as well as scan logs from a single workstation before performing the manual repair.
  7. I still see agent commands in the GSM from yesterday as "Not yet recieved"
  8. I provided an update to my ticket already logged with  Webroot Support as well as sent direct emails to my Channel Account Manager and the Sales Engineer I worked with in beta testing yesterday afternoon
At this point, I'm unable to tell how widespread the issue is. So far, it's not as broad as what we saw originally but I have multiple clients running the same EHR application that is getting quarantined.
 
Are there any additional updates or is anyone else out there experiencing the same issue?
 
Regards,
Jared

Reply