- No logs
- Kill Switch
- 6 devices
- Monthly price: $4.92
Intro to Osquery: Frequently Asked Questions for Beginners
There is a growing and passionate community around osquery, actively sharing information and perspective, answering questions, exposing challenges and dispelling misconceptions. Even so, learning the basics as you’re getting started requires a lot of piecing together bits of wisdom (ie Googling + reading + networking).
The intention of this post is to a) curate some of the great content from the community b) organize it to cover common questions for beginners c) incorporate some of what we’ve learned over the past three years through the Uptycs journey. If you like it, and it is helpful, let us know on Twitter and we’ll create a more advanced FAQ next time around.
What is osquery?
Osquery is a universal endpoint agent that was developed by Facebook in 2014. It is an active and growing open source project on GitHub, with 230 contributors and more than 90 releases to-date.
According to the official osquery docs, osquery (os=operating system) is an operating system instrumentation framework that exposes an operating system as a high-performance relational database. Using SQL, you can write a single query to explore any given data, regardless of operating system.
This is a unique approach in the security landscape, creating one agent for many operating systems, leveraging a standard query language instead of creating a proprietary one, and collecting rich data sets that have broad applications. Osquery represents a fundamental rethinking of the fragmented, siloed approach plaguing the security industry today.
With that said, osquery is just an agent—“an instrumentation framework” for data collection. Security teams looking to put osquery into production and leverage the data for security protocols will need to consider:
How you’ll configure, deploy, and manage the agent
How you’ll manage query packs (more on these below) and schedules as the community adds more
Where you’ll store osquery data (and how much it will cost)
How you’ll analyze the data—i. e., what problems are you looking to solve? What questions do you need to ask?
How you’ll handle suspicious activity that requires further investigation or remediation
Whether you need any integrations with existing tooling
How you’ll troubleshoot production issues and develop any custom functionality you may need
This often leads to a build vs buy analysis. See the section below, “What are some pros and cons of osquery? ” for additional considerations.
Learn the basics of osquery and SQL in our free training course.
What operating systems does osquery support?
Currently, osquery supports OS X (macOS), Linux, FreeBSD, and Windows. Osquery can also monitor and extract data from Docker containers. One of the most powerful features of osquery is its ability to collect and normalize relational data independent of operating system.
Because of the subtleties that exist between platforms, with other agent-based solutions users are often forced to write (and maintain) scripts to extract related information—an approach that quickly becomes a barrier to scale. Osquery solves this by exposing operating system information as normalized SQL tables. In other words, users now have the ability to ask the same questions, to get the same type of answers, regardless of operating system.
What tables does osquery support?
As of publish date, Osquery version 3. 2. 6 supports 207 tables. You can view all of the supported tables and schema information, sortable by release version, at
What type of data/information can I get from osquery?
Far richer than standard log files, here are just a few of the data elements osquery collects:
loaded kernel modules
open network connections
And a great deal more.
How much data is osquery collecting on average per endpoint each day? Where is this data stored?
We’ve observed osquery generating an average of 110MB of data per endpoint, per day. Of course, your mileage may vary depending on the monitored assets function, and what data is being collected.
Osquery supports both an interactive query console (osqueryi) and a distributed host monitoring daemon (osqueryd) that can be used to schedule queries on a reoccurring basis. From a data storage perspective, the amount of data collected, combined with the desired retention period, will ultimately dictate your requirements. For example, osquery data can be:
Not stored anywhere—i. e. queried only in real-time via osqueryi, the osquery interactive query console/shell
Stored in a SIEM (Splunk, ELK stack, etc. )
Aggregated in a security analytics platform, like Uptycs, for integrated threat intel, correlations, anomaly detection, and more
What is a query pack?
A query pack is a group, or collection of queries, designed to accomplish a particular function or task. Query packs are typically configured with a specific run schedule to help avoid any potential impact to the host machine. For example, there are query packs for incident response, vulnerability management, known OS X malware, and more (you can find a full list of query packs here).
It’s important to note that while query packs will help you collect organized sets of data, simply running a query pack for compliance, for example, does not mean you are compliant— it means you’re scheduled to collect the data required to answer questions about your compliance standing. You still need to review that data and understand what it means for your particular compliance standards and goals.
Query packs are also meant to work for the majority and require the end user to ensure usefulness, determine if any pruning is required, and weigh the performance impact. Commercial offerings (like Uptycs) provide this query pack optimization and offer the analytical insight required to take action based on the information the query pack yields (ie, X machines are failing compliance check Y because of Z. )
What are some pros and cons of osquery?
In addition to the benefits of an open source universal approach, Trail of Bits shares that teams like osquery because it’s:
Simpler to use
Exposes teams to new endpoint data to which they never before had access
Some of osquery’s cons, or areas for improvement, include requests for more extensive documentation (especially on Windows), commercially available support, and continued expansion and parity for operating systems outside of macOS and Linux. In addition, even resource-rich security teams that have deployed a fully open source/DIY solution around osquery have learned that:
Cost of data storage can be high—For example, Elastic is 3x the cost of the bytes transmitted from osquery. On average, 110MB are transmitted from each endpoint per day.
Translating the incremental data is hard—Making sense of the information for vulnerability management, threat investigation, compliance and audits, etc. is quite complicated and requires more heavy lifting than initially anticipated.
Optimizing queries and query packs is critical—Building your own query packs can have an impact on computing resources. A query will often pull far more system data than anticipated and this can cause systems to crash, yet testing and optimization before deployment is taxing, and in some cases, not possible.
Third-party data is still needed—For threat or intrusion detection, integrated third-party data sets are still necessary.
For a more technical perspective on some of the current challenges of osquery, watch this video from QueryCon 2018 where Teddy Reed, Facebook Security Engineering Manager, shares what keeps him up at night with osquery.
How do I get started and install osquery?
There are a few options for getting started with osquery. Before you begin, you’ll need to consider:
How you’ll manage query packs and schedules as the community adds more
How you’ll analyze the data—i. e., what problems are you looking to solve?
Deploying the osquery agent
Download osquery from —you’ll find macOS, Linux, RPM, Debian, and Windows versions (you may need to customize your configuration). For example, here is a post from Joshua Brower at Defensive Depth that walks through custom MSI configs.
We’ve also created a few handy videos that walk through various osquery installations. You can see them on YouTube.
Build it: You can build an end-to-end osquery solution yourself, pairing other open source and commercially available products with custom developed functionality if time, cost, and resources aren’t limiting constraints. This post from Chris Long of Palantir provides a comprehensive playbook of how they constructed a solution for rapid incident response.
Buy it: You can sign up for a free trial of Uptycs—this will enable you to deploy the necessary versions of the osquery agent. You’ll also have access to the Uptycs security analytics platform, which collects, aggregates, and analyzes osquery data for fleet visibility, intrusion detection, vulnerability management, incident investigation, and audit & compliance.
How do I ask questions or extract information from osquery?
Using SQL (Standard Query Language), and an understanding of the osquery tables where the data you require is stored, you can construct commands that return nearly any piece of information you desire about a single endpoint where osquery is running. Here is an overview of how to construct some of the most common SQL queries for osquery. If you want to query many (or all) of your endpoints at once, you’ll need additional instrumentation and a data store for aggregation across your infrastructure.
What are some basic SQL commands I need to know?
Many engineers and developers have used SQL before, but still find a refresher helpful. We also hear that the way SQL is applied and optimized for osquery is slightly different than how they used or applied SQL before. For example, joining tables, using count or limit functions and filtering results are popular given the volume of fleet-wide data you’ll be working with in osquery.
Two of the most common and basic queries are select * from uptime; and select * from users limit 5;. Check out this SQL introduction for osquery to learn common queries and how to join tables, filter results, and more.
Learn more about osquery:
Osquery: What it is, how it works, and how to use it
HTTP Rotating & Static
- 40 million IPs for all purposes
- 195+ locations
- 3 day moneyback guarantee
Installing osquery on Windows
We recommend installing on Windows using the Chocolatey package manager, or from the latest official binaries available on the Downloads page.
For those needing more customization of their deployment, the steps taken by the installation are explained in more detail, below.
Installing with Chocolatey
Each osquery tag (stable release) is published to Chocolatey for our supported versions: By default Chocolatey will install the binaries, example packs, example configuration, and an OpenSSL certificate bundle to C:Program Filesosquery and nothing more. You can pass Chocolatey the –params=’/InstallService’ flag or make use of osquery’s –install flag with C:Program Filesosqueryosqueryd –install to install a Windows SYSTEM-level service for the osqueryd daemon.
Installing osquery via the MSI package
For generating an MSI installer package, we support two methods.
The first method is with minor modifications to the CMake build steps:
First, install the Wix Toolset. With Chocolatey, choco install wixtoolset and then add C:Program Files (x86)WiX Toolset v3. 11bin to the system PATH. As of the time of this writing, the Chocolatey package installer doesn’t add this to the PATH for you; you must add it yourself.
When configuring the build, specify a version string for the osquery package using the CMake parameter -DOSQUERY_VERSION.
When building, provide an additional CMake parameter, –target package.
An example of a CMake build that generates an MSI package:
cmake -G “Visual Studio 16 2019″ -A x64 -T v141 -DOSQUERY_VERSION=”4. 0. 0”.. src
cmake –build. –config RelWithDebInfo –target package
The second method is to use the script 1 included in the source tree. This is a PowerShell script that will generate an MSI package for installing osquery. Running. toolsdeployment1 ‘msi’ from the source root will generate a standalone MSI package along with the example packs, configuration, and OpenSSL cert bundle.
To get osquery running as a SYSTEM-level service on Windows, one must ensure two things:
is running with safe permissions
The Windows Service Control Manager has all of the correct information for running the daemon
The daemon is considered safe if the binary and the directory in which the binary resides do not allow non-privileged write accesses and both are owned by either the Administrators group or the SYSTEM account.
The recommended way to set these ACLs is with PowerShell, and we’ve written a helper function to handle these permissions. To do so,. source the file and call the function, as follows:
C:UsersThorworkrepososquery [master ≡]
λ Set-SafePermissions C:Program Filesosqueryosqueryd
If you’d prefer to manually set the permissions, check the C:Program Filesosqueryosqueryd directory and ensure that no users or groups have write permissions with the exception of the Administrators group or the SYSTEM account. Read and execute permissions are expected and safe, so also ensure the Users group has both.
Now that osquery is properly laid out on the filesystem, we need to create a new Windows service to launch and manage the daemon. If you’re using Chocolatey, you can pass the –params=’/InstallService’ flag during installation to have Chocolatey set up the Windows service for you. In general, any method to install a Windows system service will suffice. You just need to ensure to specify the –flagfile option in the service binary path, and give the full paths for both the daemon binary and flag file.
To install the service using Powershell we bundle a helper function living in the repo at. tools1 which can be invoked as follows:
λ. 1 -install -startupArgs “C:Program Filesosquery”