Question

Data dictionary / field list available in API

  • 3 June 2020
  • 4 replies
  • 192 views

Still struggling a little bit trying to figure out what fields are available to be pulled via the API.  I can see some of what the template scripts are doing, but I’m trying to write a report similar to one that exists in the On Demand reports section of the WebRoot SecureAnywhere console called DNS: Active Hosts (for the last 30 days).  Right now, I’m having to manually run this report from the Webroot console and download to CSV to send to my clients as it’s not available from the Scheduled Report templates area.  I also don’t see a similar template in the Report Writer.  I literally need to pull only the following fields for each host using the DNS service:

  • Hostname
  • Username
  • Requested
  • Blocked

This needs to be in a list format and not summarized in any way.  It seems most of the templates in the report writer want to summarize or give Top 10 stats and the like.  Having trouble figuring out how to pull this data.

Any Suggestions?

Thanks!


4 replies

Userlevel 6
Badge +26

@kfranklin  - the console has basic dynamic reports for top active hosts which can show the number of Requested (usually a high number) and Blocks for quick review. You can export the “blocked” list by host to CSV. However, Requested is not exportable.
NOTE: Reason - in 7 days a single host can make a request that run well above 100K DNS requests, which
A) would take time to export and put a load on the console servers if thousands of techs made the request in similar time frame and
B) you’d have a lot of data to pull into an excel spreadsheet that is basically difficult to manage and somewhat unusable. (Not that it’s impossible, but a lot of data to handle that would make little sense.))

For performance reasons and high volume of data, we’ve determined it’s best to pull smaller data sets that are more “top categories” which concatenate the data a little more than just pulling all raw data.
NOTE: With large data sets, it could take hours to export and cause the API to time out. For example. If you pulled a report that had 10 users for 30 days and each user had 300K DNS requests (very conservative), that would be a report with well over several million rows and an extremely large file that would be nearly impossible to open or use. For this reason we’ve provided standard reports that are the top categories, but you can add more relevant categories through customization.

In the Universal Reporter, there are two starting templates for Allowed Reports: 
DNS: Allowed - Domains and Users - Using categories Finance (3) and Motor Vehicles (81)
DNS: Allowed - Domains and Users - Using categories Streaming Media (25) and Social Networking (14)
You can modify these to include additional categories that are more relevant to your environmental needs or you can copy these by following the instructions for making additional templates. Within each template, you can add category codes by editing the PS1 template on or around line 361. The category codes can be found here: http://www.brightcloud.com/tools/change-request-url-categorization.php?endpoint=webroot.com (Click on the Web Category Description link on this page and it will show all categories and their corresponding code)

You can do the same with blocked templates and combine them into one report after each have been run. Pulling data from both data sets, Requested and Blocked has performance ramifications, so it’s faster to pull each independently and combine them after the data has been exported to get closer to the data you want.

Hmmm… I’m sure you know a lot more about this than I do, however it does appear that the total DNS requests by endpoint is exportable (or I’m using the wrong terminology).  Again, in the Webroot Console: Reports, On Demand, Choose a site, DNS: Active Hosts, Last 30 days.  I see 4 columns in the output when I run it. Host name, User Name, Requested, Blocked.  All 4 of those columns seem to export to CSV just fine and very quickly.  So while I get the fact there is a lot of data behind this, somehow this view seems to summarize and display and export it very quickly.  If I could just schedule this particular report to auto-export and e-mail to a client, I’d have no work to do trying to use a different tool to get at this data.

I’ll keep digging and I appreciate the feedback and if you have any other suggestions, I’d appreciate it.

Thanks

Userlevel 6
Badge +26

@kfranklin - Yes, if all you want totals only in that Dynamic report, then that export is the only option. It will only output that data. Host Name - User - Requested (Total only) - Blocked (Total Only) - but will not export the actual DNS URL/Domains, which is what I assumed was being requested. Unfortunately, that dynamic report is not able to be scheduled at this juncture. It has been requested for development to have that included and is being considered. I do not have a time frame.

Yep, the client wanting this is not after the detail in this report.  He’s looking at getting a general overview of activity levels by machine and user.  Doesn’t need the website detail on this one.  If it could be scheduled, it would be fantastic and would save me a ton of time.

 

Thanks again!

Reply