Security Onion 2.4 Base OS

Introduction

Recent events have forced us to change course on the base operating system (OS) for Security Onion 2.4. 

On 6/21/2023, Red Hat announced changes to their source code availability for Red Hat Enterprise Linux (RHEL):

https://www.redhat.com/en/blog/furthering-evolution-centos-stream

On 6/26/2023, Red Hat then posted a follow-up:

https://www.redhat.com/en/blog/red-hats-commitment-open-source-response-gitcentosorg-changes

These announcements prompted us to go back to first principles and re-evaluate the base OS options for our upcoming Security Onion 2.4 RC1 release.

First Principles

To re-evaluate our base OS options based on first principles, we start with the basic hard requirements. Security Onion 2.4 primarily consists of Docker images orchestrated by Saltstack, so here are our requirements for the base OS:

• stable Linux kernel
• stable Docker packages
• stable Saltstack packages
• freely available at no cost
• long term support (greater than 3 years)

In addition to the requirements above, our customers have indicated certain preferences:

• Most customers prefer some sort of Red Hat derivative.
• Some customers strongly prefer operating systems that meet specific US government standards or certifications.
  
  

Options

Based on the requirements and preferences above, we considered the following options:

• Rocky Linux 9 (free RHEL rebuild) - https://rockylinux.org/ 
• Alma Linux 9 (free RHEL/CentOS rebuild) - https://almalinux.org/ 
• Oracle Linux 9 (free RHEL rebuild) - https://www.oracle.com/linux/ 
• CentOS Stream 9 (free RHEL upstream) - https://www.centos.org/centos-stream/ 
• Ubuntu 22.04 (free Debian derivative) - https://ubuntu.com/ 
• Debian 12 - https://www.debian.org/ 

Over the last few weeks, we performed an exhaustive investigation of each option to see if they satisfy the requirements and preferences above and also determine if there are any additional advantages or disadvantages.

Security Onion 2.4 ISO Image

At the conclusion of our exhaustive investigation, we decided to base our Security Onion 2.4 ISO image on Oracle Linux 9 for the following reasons:

• Oracle Linux 9 satisfies the highest number of requirements and preferences for the greatest number of customers.
• Oracle Linux is a RHEL rebuild that has been available since 2006 so they have a long track record.
• Oracle Linux is free. From https://yum.oracle.com/oracle-linux-downloads.html:
  Since 2006, Oracle Linux has been completely free to download and use. Free source code, binaries, and updates. Freely redistributable. Free for production use.
• Oracle is well funded and fully committed to Oracle Linux in the long term:
  https://www.oracle.com/news/announcement/blog/keep-linux-open-and-free-2023-07-10/ 
• Oracle Linux 9 FIPS (Federal Information Processing Standards) certification is listed as “Under Test” as of 3/7/2023:
  https://csrc.nist.gov/Projects/cryptographic-module-validation-program/modules-in-process/IUT-List 
• Oracle Linux offers the ability to run a newer Linux kernel which, in theory, should be more secure than the default kernel included in other RHEL rebuilds:
  https://unix.foo/posts/enterprise-linux/
  https://blogs.oracle.com/linux/post/tracking-linux-stable-kernels-with-uek 
  
  

Network Installation

If you don’t want to use our Security Onion 2.4 ISO image, you can still perform a network installation of our Security Onion components after manually installing one of the following:

• Oracle Linux 9
• Rocky Linux 9
• Alma Linux 9
• CentOS Stream 9
• RHEL 9
• Ubuntu 22.04
• Debian 12
  
  

Support

Customers with premium support and professional services can reach out to their normal support contacts for more information about support. 

If you are a non-paid community user, then please pay close attention to the support levels below.

Supported

Our Security Onion 2.4 ISO image (based on Oracle Linux 9) is the only fully supported installation method. Choose this option if any of the following apply to you:

• You are deploying in an enterprise environment.
• You are deploying in an airgap environment.
• You are performing a distributed deployment.
• You want the quickest and easiest installation with the fewest issues.
• You need full support.
  
  

Unsupported

If you don’t want to use our Security Onion 2.4 ISO image and choose to perform a manual OS installation followed by a network installation of our Security Onion components, then we recommend using Oracle Linux 9 or Rocky Linux 9. CentOS Stream 9 or Alma Linux 9 should also work. Another option might be RHEL 9 itself although that is a paid option.

If you really want to run Ubuntu 22.04 or Debian 12, then please note that these distros may work but they get less testing and therefore you will be more likely to run into issues.

Q&A

What will the Security Onion 2.4 ISO image be based on?

Our Security Onion 2.4 ISO image will be based on Oracle Linux 9.

Why Oracle Linux?

Oracle Linux has been around since 2006 and so it has a long track record. Additionally, FIPS certification is in progress. Finally, Oracle Linux offers a newer Linux kernel which, in theory, should be more secure than the default kernel included in other RHEL rebuilds.

Is Oracle Linux free?

Yes, from https://yum.oracle.com/oracle-linux-downloads.html:

Since 2006, Oracle Linux has been completely free to download and use. Free source code, binaries, and updates. Freely redistributable. Free for production use.

If we don’t want to use the Security Onion 2.4 ISO image or Oracle Linux, do we have other options?

If you don’t want or need support, then you can choose from Rocky Linux 9, Alma Linux 9, CentOS Stream 9, RHEL 9, Ubuntu 22.04, or Debian 12. However, please note that if you choose one of these options there will be more manual work required and you may be more likely to run into issues.

Why not use Rocky Linux 9 for the Security Onion 2.4 ISO image?

As of 7/25/2023, Rocky Linux 9 is not yet listed at the FIPS certification pages:

https://csrc.nist.gov/Projects/cryptographic-module-validation-program/modules-in-process/IUT-List

https://csrc.nist.gov/Projects/cryptographic-module-validation-program/modules-in-process/Modules-In-Process-List 

Why not use Alma Linux 9 for the Security Onion 2.4 ISO image?

Alma Linux does not yet have FIPS certification. Also, it has only been around since 2021 so it doesn’t have that long of a track record.

Why not use CentOS Stream for the Security Onion 2.4 ISO image?

CentOS Stream does not have any FIPS certification whatsoever.

Why not use Ubuntu for the Security Onion 2.4 ISO image?

Standard Ubuntu has no FIPS certification. FIPS certification requires a paid upgrade to Ubuntu Pro. Additionally, Ubuntu seems to be focused on their own snap architecture for the future.

Why not use Debian for the Security Onion 2.4 ISO image?

Debian does not have any FIPS certification whatsoever.

Why not use another Linux distro like fill in the blank?

We considered several other Linux distributions but only the ones listed above met the core requirements.

When do these Security Onion changes take effect?

These changes go into effect for the upcoming Security Onion 2.4 RC1 release.

When will Security Onion 2.4 reach General Availability (GA)?

These OS changes delayed our release schedule for Security Onion 2.4, but it was important to take our time and fully investigate our options. Security Onion 2.4 RC1 is coming soon. Stay tuned!

What does all of this mean for Security Onion 2.3?

There are no planned OS changes for 2.3.

I am a current customer with premium support and professional services and I have other questions about this change. To whom should I reach out?

Please feel free to reach out to your account manager.

I am a community user of Security Onion and I have other questions about this change. How may I ask those questions?

You may start a new discussion at https://securityonion.net/discuss. 

UPDATE 2023/07/27 - Added that these changes go into effect for the upcoming Security Onion 2.4 RC1 release.

Red Hat, Rocky Linux, and Security Onion

On 2023/06/21, Red Hat announced a change to their source code access:
https://www.redhat.com/en/blog/furthering-evolution-centos-stream

What does this change mean for Security Onion?

First, this change should have no effect on the current Security Onion 2.3 platform. 

For Security Onion 2.4, our plan is to use Rocky Linux as the base platform. On 2023/06/22, Rocky Linux posted the following: https://rockylinux.org/news/2023-06-22-press-release/

Based on Rocky's announcement, we are optimistic that we can continue our plans to use Rocky Linux. If, for some reason, this changes we have contingency plans available.

We are monitoring this situation closely and will provide further updates as needed.

UPDATE 2023/06/25 Here's an additional update from the Rocky Linux team:
https://rockylinux.org/news/brave-new-world-path-forward/

UPDATE 2023/07/25 We've published an updated blog post:
https://blog.securityonion.net/2023/07/security-onion-24-base-os.html

Security Onion Hybrid Hunter 1.0.1 Tech Preview Available for Testing!

From Doug Burks:

When Mike Reeves joined Security Onion Solutions in January 2018, one of the first things we discussed was building a brand new Security Onion platform with the following characteristics:

• Move from Ubuntu DEB packages to Docker images
• Support both Ubuntu 16.04 and RedHat/CentOS 7
• Higher performance
• More centralized configuration

In just a few short months, Mike has done an incredible amount of work to make this idea a reality and we announced it at Security Onion Conference 2018:

https://www.youtube.com/watch?v=MVZ33P_tN-g

Here’s Mike Reeves to tell you more about this exciting platform!

From Mike Reeves:

First off I would like to thank everyone who presented at or attended the 2018 Security Onion Conference. This was the best one yet and I am already excited about next year. I wanted to take the time to talk about some of the long term plans we have for the Security Onion platform and how these potential changes, which we’ve code named “Hybrid Hunter”, may affect your deployment. 

The general theme of Hybrid Hunter is simplification. We want you spending more time finding evil than running your sensor grid. Since 2008, Security Onion’s primary mission was to provide a Network Security Monitoring distribution that could be deployed in minutes instead of days or weeks.  Hybrid Hunter expands on this and allows it to scale better in large enterprise networks. 

At Security Onion Con 2018, Doug and I unveiled some details behind Hybrid Hunter. We received so much feedback and we are very appreciative to all of you. One item of feedback I received involved changes to the way Security Onion operates today. I think a perfect use case we can use to illustrate the changes is Logstash. Today, when there is an update to Logstash a couple of things happen. First, the Docker container gets replaced with a container running a newer version of Logstash. Additionally, an Ubuntu package is downloaded which updates the Logstash configuration, e.g., parsers, output configurations, etc. If we continued this method and wanted to support RedHat/CentOS, we would need to create a separate package to manage the parsers. Multiply that effort by over fifty packages, along with nuanced differences between the operating systems, and we would have an arduous task!

Our intent is for Hybrid Hunter to deliver as many components as possible as Docker containers. Gone would be the days where a new DEB or RPM package would be required for delivery of these changes, thus allowing us to support multiple Linux distributions going forward. Updating most Security Onion components would be as easy as updating Logstash and other Docker containers today. The process of updating would also allow for easy rollback. If something doesn’t work properly, the container can simply be stopped and the older version applied. The administrator will still run “soup”; however, it would not apply packages for SO components, just Docker containers!

For those of us that like to get our hands dirty when it comes to tweaking, you will be glad to know that the configurations will be centralized in the new platform. Today you have to visit multiple config files in multiple places to do tuning. Our goal is to put as much of this as possible into a single location, allowing you to tune more in less time. 

Even though there are some new tools being added or replaced, the end user experience should remain the same. The training you get from Security Onion Solutions will be applicable to the current version and Hybrid Hunter, with minor differences for advanced tuning. You will still pivot to PCAP the same way even though Google Stenographer will be gathering the packets instead of netsniff-ng. The whole reason for this change is to get more consistent results when pulling PCAP but it doesn’t change the way you use SO. The end result is the same PCAP with the same experience. Changing from PF_RING to AF_PACKET improves the way that we acquire packets but does not change the end result of what you will see in the console. AF_PACKET allows you to expand your tuning possibilities with Suricata and improves performance. Those alerts will still look the same and will be more consistent. Zeek (formerly Bro) will see a performance improvement over using PF_RING but the meta data will look the same.  We will also be allowing our users to select Community Bro if they so choose. Either choice will provide the same great metadata you have seen in Security Onion for years … and more!

I would also like to reiterate that there is no firm release date set. We are gathering input from you, the community, on other ways to make SO easier to deploy and tune. Our goal is to make the most successful experience for our users and expand our capabilities to fit the enterprise security monitoring needs of customers of all sizes.

Thanks,

Mike Reeves

Product Manager

Security Onion Solutions     

Try It Out

Try out the Hybrid Hunter Tech Preview here:

https://github.com/Security-Onion-Solutions/securityonion-saltstack/blob/master/README.md

Feedback

If you have questions, problems, or other feedback regarding Hybrid Hunter, please post to our subreddit and prefix the title with [Hybrid Hunter]:

https://www.reddit.com/r/securityonion/

FAQ

Is the current Ubuntu-based platform still supported?

Yes, the current Ubuntu-based platform is still fully supported.  Once the new Hybrid Hunter platform reaches final release, we will announce plans to migrate from the current Ubuntu-based platform to the new platform.

Why the change from Ubuntu DEB packages to Docker images?

Docker images are easier to build and maintain and allow us to support other distros like CentOS.

Why the change from PF_RING to AF_PACKET?

AF_PACKET is included in the Linux kernel itself and thus doesn't require a separate kernel module.  It also provides some additional tuning capability.

Why manage everything with salt?

Salt will allow us to manage configuration centrally on the master node so that it won't matter whether you have 1 box or 100, you can still manage everything easily from a central location.