Friday, June 2, 2017

Towards Elastic on Security Onion: Technology Preview 2 (TP2)

UPDATED 2018/04/09! We've released a newer version!

We recently announced our move towards the Elastic stack:

In the last few weeks, we've made tremendous progress, so it's time for our second technology preview (TP2)!

Changes from the last Technology Preview
  • upgraded from Elastic 2.4.4 to 5.4.0
  • Elasticsearch, Logstash, and Kibana each run in their own Docker containers
  • lots more dashboards
  • new Logstash parsers to support more log types
  • IPv6 support
  • experimental script to migrate data from ELSA to Elastic
  • Squert now leverages the same single sign on as Kibana and CapMe

Warnings and Disclaimers
  • If this breaks your system, you get to keep both pieces!
  • This script is a work in progress and is in constant flux.
  • This script is intended to build a quick prototype proof of concept so you can see what our ultimate Elastic configuration might look like.  This configuration will change drastically over time leading up to the final release.
  • Do NOT run this on a system that you care about!
  • Do NOT run this on a system that has data that you care about!
  • This script should only be run on a TEST box with TEST data!
  • This script is only designed for standalone boxes and does NOT support distributed deployments.
  • Use of this script may result in nausea, vomiting, or a burning sensation.

Enough disclaimers?  Let's do this!

Start with a disposable TEST VM with the following minimum requirements:

  • 2 CPU cores
  • 8GB RAM
  • 20GB virtual hard drive
  • (1) management interface with full Internet access
  • (1) sniffing interface (separate from management interface)
  • Security Onion ISO image installed
  • Setup ran in Evaluation Mode
Download the script:
Run the script with sudo privileges:
sudo bash
Please read through all the WARNINGS and DISCLAIMERS and ONLY proceed if you agree.

The script will take at least 10 minutes depending on the speed of your hardware and Internet connection.  At the end of the script, it will prompt you to access Kibana via the following URL:

You should then see our new Security Onion login window.  Enter the same credentials that you use to login to Sguil/Squert.  This login window will provide single sign on for Kibana, Squert, and CapMe to allow seamless pivoting to full packet capture!

Once logged into Kibana, you will automatically start on our Overview dashboard and you will see links to other dashboards as well.  As you search through the data in Kibana, you should see Bro logs,  syslog, and Snort alerts.  Logstash should have parsed out most fields in most Bro logs and Snort alerts.  Notice that the search panels at the bottom of the dashboards display the source_ip and destination_ip fields with hyperlinks.  These hyperlinks will take you to a dashboard that will help you analyze the traffic relating to that particular IP address.  UID fields are also hyperlinked.  Clicking on a UID hyperlink will start a new Kibana search for that particular UID.  In the case of Bro UIDs this will show you all Bro logs related to that particular connection.  Each log entry also has an _id field that is hyperlinked.  This hyperlink will take you to CapMe, allowing you to request full packet capture for any arbitrary log type!  This assumes that the log is for tcp or udp traffic that was seen by Bro and Bro recorded it correctly in its conn.log.

Previously, in Squert and Sguil, you could pivot from an IP address to ELSA.  Those pivots have been removed and replaced with a pivot to Kibana.

For screenshots, please see the Screenshot Tour at the bottom of this post.

For the current TODO list, please see:

We're releasing this now because we want to get your feedback as early as possible in this project.  Please try it out and send your feedback to our mailing list:

What do you think?

What works well?

What needs to be improved?

Any questions or other comments?

Thanks in advance for any and all feedback!

Want to show your support for Security Onion?
Security Onion t-shirts are available in our CafePress store!

Need Security Onion Training?
We offer both onsite and online training (although please note that Elastic will not be added to training classes until we reach a stable release):

Our annual Security Onion Conference will be Friday September 15, 2017:

Hope to see you there!

Screenshot Tour

Overview Dashboard

Bro Notices Dashboard

HIDS Alerts Dashboard

NIDS Alerts Dashboard 

NIDS alerts now contain the rule that generated the alert

Bro Connections Dashboard

Bro Connections Traffic Map

IPv6 Support

Bro DCE/RPC Dashboard

Bro DHCP Dashboard

Bro DNP3 Dashboard

Bro DNS Dashboard 

Bro Files Dashboard

Bro FTP Dashboard

Bro HTTP Dashboard

Bro Intel Dashboard

Bro IRC Dashboard

Bro Kerberos Dashboard 

Bro Modbus Dashboard

Bro MySQL Dashboard

Bro NTLM Dashboard

Bro PE Dashboard

Bro RADIUS Dashboard

Bro RDP Dashboard

Bro RFB Dashboard

Bro SIP Dashboard

Bro SMB Dashboard

Bro SMTP Dashboard

Bro SNMP Dashboard

Bro Software Dashboard

Bro SSH Dashboard

Bro SSL Dashboard

Bro Tunnels Dashboard

Bro Weird Dashboard

Bro X.509 Dashboard

Host Logs Dashboard

Stats Dashboard

Each Dashboard has a search panel with important fields hyperlinked

Clicking the Source IP hyperlink takes you to the Indicator Dashboard searching for the Source IP

Click the Destination IP hyperlink takes you to the Indicator Dashboard searching for the Destination IP

Clicking the UID hyperlink takes you to the Indicator Dashboard searching for that UID

Clicking the _ID hyperlink takes you to CapMe for full packet capture

Clicking the Squert link in Kibana takes you directly to Squert thanks to Single Sign On

SSO allows you to pivot seamlessly from Squert to CapMe for full packet capture

New scripts to manage Elastic stack

No comments:

Search This Blog

Featured Post

Sneak Peek: New Detections Feature coming in Security Onion 2.4.70!

Our latest video is a sneak peek at a NEW feature coming to our FREE and OPEN Security Onion platform in the upcoming 2.4.70 release! This n...

Popular Posts

Blog Archive