Tuesday, February 20, 2018

Security Onion Elastic Stack Release Candidate 2 and Security Onion 14.04.5.8 ISO Image!

UPDATED 2018/04/09! We've released a newer version!
https://blog.securityonion.net/2018/04/security-onion-elastic-stack-general.html

We're excited to announce that our Elastic Stack integration has now reached Release Candidate 2 (RC2)!  RC2 includes a new 14.04.5.8 ISO image that contains these RC2 components and all the latest Ubuntu and Security Onion updates as of February 14, 2018!


RC2 Highlights

Major Changes in this release:

- All Ubuntu and Security Onion updates as of 2018/2/14

- Elastic Stack Release Candidate 2:
https://github.com/Security-Onion-Solutions/security-onion/issues/1198

- Docker images based on Elastic Stack 6.1.3

- Experimental Setup has been redesigned to add new options and make them easier to understand

- If choosing Production Mode, Setup will ask if you are building a new deployment or joining an existing one:


- If building a new deployment, Setup will ask if you want to store logs locally on the master server  or output to separate load-balanced storage nodes:


- If joining an existing deployment, Setup will then ask what type of node you want to add - Forward, Heavy, or Storage:


- Forward Nodes generate and collect logs and forward them to the master server. Full packet capture remains on Forward Nodes. They do not run any Elastic stack processes. Forward Nodes require less hardware than Heavy Nodes, but typically use more network bandwidth.

- Heavy Nodes generate and collect logs and store them locally (similar to traditional ELSA sensors). Heavy Nodes run Elasticsearch and Logstash and the master server queries Heavy Nodes via cross cluster search. Heavy Nodes require more hardware than Forward Nodes, but typically use less network bandwidth.

- Storage Nodes do not generate logs themselves but simply extend the storage of the master server. They require that the master server has already been configured to load balance to additional storage nodes via redis. Storage Nodes run Elasticsearch and Logstash. The master server queries storage nodes via cross cluster search.

- so-elastic-start-logstash checks for LOGSTASH_OUTPUT_REDIS=yes in /etc/nsm/securityonion.conf and, if set, will automatically start redis and configure Logstash to output to it.  So if you build a master server and initially choose to store logs locally, you can later convert it to redis output by setting LOGSTASH_OUTPUT_REDIS="yes" in /etc/nsm/securityonion.conf and then restarting Logstash:
sudo docker stop so-logstash
sudo docker rm so-logstash
sudo so-elastic-start-logstash
- so-elastic-configure now configures Bro to output in JSON

- Logstash parsers have been updated to detect Bro logs in JSON and parse as such, otherwise fall back to csv or grok where appropriate

- Bro logs now write Bro timestamp to `@timestamp` field (Logstash timestamp can now be found in `timestamp`)

- New so-autossh-* scripts control the autossh tunnel that connects nodes to master server

- On nodes where Elasticsearch is enabled, curator should be fully configured to close indices after 30 days (see /etc/curator/ and /etc/cron.d/curator-close)

- Kibana - Management - Index Patterns should show the following Index Patterns (no more duplicates):
*:logstash-*
*:elastalert_status*
*:logstash-beats-*
- Replicas now disabled properly for all Elasticsearch indices so all indices should show as GREEN:
curl http://localhost:9200/_cat/indices

- so-migrate-elsa-to-elastic has been renamed to so-elsa-export and just exports data without automatically importing to Logstash

- Lots of other improvements and bug fixes!

Issues Resolved
Issue 1198: Elastic Stack Release Candidate 2
https://github.com/Security-Onion-Solutions/security-onion/issues/1198

Issue 1199: 14.04.5.8 ISO image
https://github.com/Security-Onion-Solutions/security-onion/issues/1199

This new ISO image has been tested by Wes Lambert, Audrius J, and Jay Hawk.  Thanks, all!

Known Issues
If you start with Bro logs in JSON and then revert back to TSV format, some fields may conflict which may result in logs no longer loading properly.  This will be resolved in RC3.  We recommend most folks keep Bro logs in JSON format for better performance.

If you refresh the field list in Kibana, you may see a conflict for the timestamp or other fields and this may result in logging or dashboard issues.  This will be resolved in RC3.  We recommend that you avoid refreshing the field list unless absolutely necessary.

For other known issues, please see the todo list for our next release:
https://github.com/Security-Onion-Solutions/security-onion/issues/1208

Thanks
Special thanks to the following for their contributions to our Elastic Stack integration!

  • Elastic.co
  • Justin Henderson
  • Mark Baggett

New Installations
We've updated the Verify_ISO page for the new ISO image:
https://github.com/Security-Onion-Solutions/security-onion/blob/master/Verify_ISO.md

Please remember to verify the signature of the downloaded ISO image using the instructions on that page.

Please note! This ISO image includes the EXPERIMENTAL Elastic stack!

The Elastic components are included in the ISO image and Setup gives you an option of Stable Setup (ELSA) or Experimental Setup (Elastic). If you do not want to try the new Elastic stack, you can choose Stable Setup.  If you choose Experimental Setup, the usual disclaimers and warnings apply!

  • Experimental Setup is BLEEDING EDGE and TOTALLY UNSUPPORTED!
  • If this breaks your system, you get to keep both pieces!
  • This is a work in progress and is in constant flux.
  • This is intended to build a quick prototype proof of concept so you can see what our ultimate Elastic configuration might look like.  This configuration will change drastically over time leading up to the final release.
  • Do NOT run this on a system that you care about!
  • Do NOT run this on a system that has data that you care about!
  • This should only be run on a TEST box with TEST data!
  • Experimental Setup may result in nausea, vomiting, or a burning sensation.

For more about Elastic Release Candidate 2, please see https://securityonion.net/wiki/elastic and the Screenshot tour at the bottom of this blog post.

Please note the following minimum hardware requirements for the Elastic stack:

  • 2 CPU cores
  • 8GB RAM

If you would prefer an ISO image with no Elastic components at all, you have a few options:

  • Install the older Security Onion 14.04.5.2 ISO image and then run "sudo soup"

OR


Existing Deployments
If you have existing ELSA installations based on a previous 14.04 ISO image, there is no need to download this new ISO image.  You can simply continue using our standard update process to install updated packages as they are made available:
https://github.com/Security-Onion-Solutions/security-onion/wiki/Upgrade

If you have existing Elastic installations (Technology Previews, Alpha, Beta, or RC1), we don't officially support upgrading to newer releases, but you can try the steps listed here:
https://securityonion.net/wiki/elastic-rc2

Release Notes
For more information about this release, please see:
https://github.com/Security-Onion-Solutions/security-onion/wiki/Security-Onion-14.04-Release-Notes

Feedback
We want to hear from you!  What works well?  What could be improved?  Please send feedback to our mailing list and include "Elastic RC2" in the Subject:
https://github.com/Security-Onion-Solutions/security-onion/wiki/MailingLists

Previous Releases
To see our progress over the last few months, please see the previous announcements:
http://blog.securityonion.net/2017/03/towards-elk-on-security-onion.html
http://blog.securityonion.net/2017/06/towards-elastic-on-security-onion.html
http://blog.securityonion.net/2017/07/towards-elastic-on-security-onion.html
http://blog.securityonion.net/2017/09/elastic-stack-alpha-release-and.html
http://blog.securityonion.net/2017/11/elastic-stack-beta-release-and-security.html
http://blog.securityonion.net/2017/11/elastic-stack-beta-2-release-and.html
http://blog.securityonion.net/2017/12/security-onion-elastic-stack-beta-3.html
http://blog.securityonion.net/2018/01/security-onion-elastic-stack-release.html

Training
We offer onsite and online training!  For more information, please see:
https://securityonionsolutions.com

Support
Need support?  Please see:
https://securityonion.net/wiki/Support

Screenshot Tour

Security Onion 14.04.5.8 20180214

Welcome to Setup

Network Configuration

Stable Setup (ELSA) vs Experimental Setup (Elastic)

Elastic Setup

Evaluation Mode vs Production Mode

Monitor (Sniffing) Interface Selection

Creating First User Account

Creating Password

Confirming Password

Confirming Options

Setup Complete

Single Sign On (SSO) for Squert, CapMe, and Kibana

Squert

CapMe

Kibana Home (Overview) Dashboard

Help

Bro Notices

ElastAlert

OSSEC HIDS Alerts

NIDS Alerts from Snort or Suricata

Bro - Connections

Bro - DCE/RPC

Bro - DHCP

Bro - DNP3

Bro - DNS

Bro - Files

Bro - FTP

Bro - HTTP

Bro - Intel

Bro - IRC

Bro - Kerberos

Bro - Modbus

Bro - MySQL

Bro - NTLM

Bro - PE

Bro - RADIUS

Bro - RDP

Bro - RFB

Bro - SIP

Bro - SMB

Bro - SMTP

Bro - SNMP

Bro - Software

Bro - SSH

Bro - SSL

Bro - Syslog

Bro - Tunnels

Bro - Weird

Bro - X.509

Autoruns

Beats

OSSEC Logs

Sysmon

Domain Stats

Firewall

Frequency Analysis

Stats

Syslog

No comments:

Search This Blog

Featured Post

1-month End Of Life (EOL) reminder for Security Onion 2.3

In October of last year, we announced the End Of Life (EOL) date for Security Onion 2.3: https://blog.securityonion.net/2023/10/6-month-eol-...

Popular Posts

Blog Archive