Security Onion: Security Onion Elastic Stack Release Candidate 2 and Security Onion 14.04.5.8 ISO Image!

UPDATED 2018/04/09! We've released a newer version!
https://blog.securityonion.net/2018/04/security-onion-elastic-stack-general.html

We're excited to announce that our Elastic Stack integration has now reached Release Candidate 2 (RC2)! RC2 includes a new 14.04.5.8 ISO image that contains these RC2 components and all the latest Ubuntu and Security Onion updates as of February 14, 2018!

RC2 Highlights

Major Changes in this release:

- All Ubuntu and Security Onion updates as of 2018/2/14

- Elastic Stack Release Candidate 2:
https://github.com/Security-Onion-Solutions/security-onion/issues/1198

- Docker images based on Elastic Stack 6.1.3

- Experimental Setup has been redesigned to add new options and make them easier to understand

- If choosing Production Mode, Setup will ask if you are building a new deployment or joining an existing one:

- If building a new deployment, Setup will ask if you want to store logs locally on the master server or output to separate load-balanced storage nodes:

- If joining an existing deployment, Setup will then ask what type of node you want to add - Forward, Heavy, or Storage:

- Forward Nodes generate and collect logs and forward them to the master server. Full packet capture remains on Forward Nodes. They do not run any Elastic stack processes. Forward Nodes require less hardware than Heavy Nodes, but typically use more network bandwidth.

- Heavy Nodes generate and collect logs and store them locally (similar to traditional ELSA sensors). Heavy Nodes run Elasticsearch and Logstash and the master server queries Heavy Nodes via cross cluster search. Heavy Nodes require more hardware than Forward Nodes, but typically use less network bandwidth.

- Storage Nodes do not generate logs themselves but simply extend the storage of the master server. They require that the master server has already been configured to load balance to additional storage nodes via redis. Storage Nodes run Elasticsearch and Logstash. The master server queries storage nodes via cross cluster search.

- so-elastic-start-logstash checks for LOGSTASH_OUTPUT_REDIS=yes in /etc/nsm/securityonion.conf and, if set, will automatically start redis and configure Logstash to output to it. So if you build a master server and initially choose to store logs locally, you can later convert it to redis output by setting LOGSTASH_OUTPUT_REDIS="yes" in /etc/nsm/securityonion.conf and then restarting Logstash:

sudo docker stop so-logstash
sudo docker rm so-logstash
sudo so-elastic-start-logstash

- so-elastic-configure now configures Bro to output in JSON

- Logstash parsers have been updated to detect Bro logs in JSON and parse as such, otherwise fall back to csv or grok where appropriate

- Bro logs now write Bro timestamp to `@timestamp` field (Logstash timestamp can now be found in `timestamp`)

- New so-autossh-* scripts control the autossh tunnel that connects nodes to master server

- On nodes where Elasticsearch is enabled, curator should be fully configured to close indices after 30 days (see /etc/curator/ and /etc/cron.d/curator-close)

- Kibana - Management - Index Patterns should show the following Index Patterns (no more duplicates):

*:logstash-*
*:elastalert_status*
*:logstash-beats-*

- Replicas now disabled properly for all Elasticsearch indices so all indices should show as GREEN:
curl http://localhost:9200/_cat/indices

- so-migrate-elsa-to-elastic has been renamed to so-elsa-export and just exports data without automatically importing to Logstash

- Lots of other improvements and bug fixes!

Issues Resolved
Issue 1198: Elastic Stack Release Candidate 2
https://github.com/Security-Onion-Solutions/security-onion/issues/1198

Issue 1199: 14.04.5.8 ISO image
https://github.com/Security-Onion-Solutions/security-onion/issues/1199

This new ISO image has been tested by Wes Lambert, Audrius J, and Jay Hawk. Thanks, all!

Known Issues
If you start with Bro logs in JSON and then revert back to TSV format, some fields may conflict which may result in logs no longer loading properly. This will be resolved in RC3. We recommend most folks keep Bro logs in JSON format for better performance.

If you refresh the field list in Kibana, you may see a conflict for the timestamp or other fields and this may result in logging or dashboard issues. This will be resolved in RC3. We recommend that you avoid refreshing the field list unless absolutely necessary.

For other known issues, please see the todo list for our next release:
https://github.com/Security-Onion-Solutions/security-onion/issues/1208

Thanks
Special thanks to the following for their contributions to our Elastic Stack integration!

Elastic.co
Justin Henderson
Mark Baggett

New Installations
We've updated the Verify_ISO page for the new ISO image:
https://github.com/Security-Onion-Solutions/security-onion/blob/master/Verify_ISO.md

Please remember to verify the signature of the downloaded ISO image using the instructions on that page.

Please note! This ISO image includes the EXPERIMENTAL Elastic stack!

The Elastic components are included in the ISO image and Setup gives you an option of Stable Setup (ELSA) or Experimental Setup (Elastic). If you do not want to try the new Elastic stack, you can choose Stable Setup. If you choose Experimental Setup, the usual disclaimers and warnings apply!

Experimental Setup is BLEEDING EDGE and TOTALLY UNSUPPORTED!
If this breaks your system, you get to keep both pieces!
This is a work in progress and is in constant flux.
This is intended to build a quick prototype proof of concept so you can see what our ultimate Elastic configuration might look like. This configuration will change drastically over time leading up to the final release.
Do NOT run this on a system that you care about!
Do NOT run this on a system that has data that you care about!
This should only be run on a TEST box with TEST data!
Experimental Setup may result in nausea, vomiting, or a burning sensation.

For more about Elastic Release Candidate 2, please see https://securityonion.net/wiki/elastic and the Screenshot tour at the bottom of this blog post.

Please note the following minimum hardware requirements for the Elastic stack: