Chapter 1: Why are containers important?
As with many things in the Technology space, Containers are a buzz word that draws a lot of attention. Cloud, AI, Agile, DevOps, and more have all gone through massive attention, and rapid adoption sprees that transform the way we do business and often for good reason. Containers have been continuously gaining traction for sense about 2013 when Docker was released to the general public as an open source tool but why?
Pets vs Cattle
The problem
Historically a system or server that hosts an application such as a website or service needs to be installed, maintained, patched, debugged, and generally cared for. Failure to address these sort of things will inevitably compromise security and even by applying regular updates, things change and legacy applications will break. Everybody knows a time when an application suddenly stopped working because some patch was pushed or some dependent application was phased out.
Virtual Machines can fix some of these issues by letting us create many more systems on a single piece of hardware. This way if some legacy program needs Java 8, that machine can have that version while another virtual machine can have a more modern Java 21 but this compounds the issue of maintainability and increases the burden of updates, and decreases the overall system security!
Traditional systems (VMs included) ultimately become very unique. They are manually cared for across multiple years and ultimately the process of setting them up becomes a forgotten art because the SME who configured it moved on, or perhaps just forgot what they had to do. These traditional systems are one of a kind and make it very difficult to replicate when a company grows or a new project appears. The administrators put a lot of blood sweat and tiers into the systems taking days to create or train them to do a task, making them a permanent member of their infrastructure, and often giving them a name... essentially treating the system as if it were a pet!
The DevOps/Cloud-Friendly/Container solution
Containers focus the purpose of a server, the application itself. This could be the web service, a database, or any other application. They effectively "Contain" exactly what that application needs to run and nothing else by simply including any libraries lib and binary applications bin that the software depends on. This way when an application needs to be deployed on another piece of hardware, everything that application needs is packaged inside the container and comes along with it. It no longer matters that the version of Java is different from another application, or how another application is configured because containers are also isolated form one another!
The popular docker logo actually includes images of shipping containers because much like how shipping containers revolutionized logistics by standardizing a size and shape that could fit on trains, ships and trucks, containers standardize software deployments! It no longer matters if the application is running on your dusty old laptop, someone's bleeding edge MacBook Pro or even in the cloud because everything the application needs between all of these environments has stayed the same.
Developers don't need to concern themselves with nearly as many dependencies of the system, freeing them up to focus on maintaining the application itself, improving security by updating the application or dependencies, easily testing the container on an entirely different machine and knowing for certain that it will run the same way everywhere. These are the foundational concepts of DevOps and drastically improved the productivity of the industry and changed the way we view our servers that host applications to be more in line with how farmers treat cattle.
- Pets
- Given names
- They are unique, lovingly handled, raised and cared for
- When they get ill, you nurse them back to health
- Cattle
- Given numbers instead of names
- They are almost identical to other cattle
- When they get ill, you replace them with another one.
Where OCI containers fit
There are multiple kinds of containers and we will cover some of them in this course but the predominant type of container are those defined by the Open Container Initiative (OCI). Docker is one such compliant container tool, as are Podman, Kubernetes, OpenShift, Cri-O, many cloud offerings and more! This type of container was largely built for the cloud meaning that they are hosted on machines without a dedicated monitor, keyboard and mouse. as a result they are usually web based applications that are hosted in Linux environments.
- OCI Complaint containers work well when:
- The application does not have a non-web based GUI.
- The application runs on Linux and can be repeatably installed/configured with a relatively small number of dependent libraries.
- The application is ephemeral (designed to be recover easily) and can be easily replaced by another instance of the same application.
- The application follows a microservices style design where each portion of the application aims to do one thing and do it well.
- The use case requires repeatable deployments across a range of systems (cloud, VMs, bare metal, etc.)
- OCI Containers do not fit when:
- The application requires a non-web based user interface
- The application runs on multiple operating system or kernels (IE Windows)
- The application is monolithic, large or generally slow to start
- Your use case required near real time compute speeds
Container Images vs containers
Containers in general are designed to be ephemeral and leverage design that certifies a 'golden copy' of the container through the container image. A container image is usually what we are referencing when we talk about what is included within a container and can be thought of like an old CD ROM. The image cannot be modified once it is built and has everything needed to run the application within it. Any updates to the application or libraries/binaries within the image will require you to build a new image to replace the old one. You can think of an image as the 'blueprint' for building a container.
Containers themselves are actually spawned from the image and can be very quickly created and deleted without loss of the image content. However, it is important to note that any changes that were made to the container while it was running will be lost unless you explicitly store things (covered later in this course.). A single container image can be used to create countless containers and one of the neatest benefits is that the container only ever tracks the differences between itself and the image which is usually very minimal (in the oder of kilobytes.)
Demonstration
The following was run on a "Cloud Native Kubernetes" instance on ACloudGuru
cloud_user@912cec8e983c:~$ docker image pull ubuntu Using default tag: latest latest: Pulling from library/ubuntu 3153aa388d02: Pull complete Digest: sha256:0bced47fffa3361afa981854fcabcd4577cd43cebbb808cea2b1f33a3dd7f508 Status: Downloaded newer image for ubuntu:latest cloud_user@912cec8e983c:~$ docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE ubuntu latest 5a81c4b8502e 4 weeks ago 77.8MB cloud_user@912cec8e983c:~$ docker run --detach ubuntu sleep 5000 244812aab66aec47fe08455acf15757110e09a56fb6bc568af346ea84bb76397 cloud_user@912cec8e983c:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 244812aab66a ubuntu "sleep 5000" 4 seconds ago Up 3 seconds goofy_haibt cloud_user@912cec8e983c:~$ docker run --detach ubuntu sleep 5000 ac7eb755ec0754446048e424f5b88b287d7f87f643b28cb84673cc4b917053c6 cloud_user@912cec8e983c:~$ docker run --detach ubuntu sleep 5000 acd064dcd0eee6e9867af764028dc4426c17c5adf988ea349e22ee3155f8f442 cloud_user@912cec8e983c:~$ docker run --detach ubuntu sleep 5000 d24d261e2003998b91a641cbc9aec71ecaa23fad048531368eadaa77319e215f cloud_user@912cec8e983c:~$ docker run --detach ubuntu sleep 5000 705488122cfc5a43e4437b1384476a78bd9eadc25afb852d1ee6006db30324bb cloud_user@912cec8e983c:~$ docker run --detach ubuntu sleep 5000 1d9227acfe1cea0debc48d494cff9937ea58007353f54a335801b54e9eae1e79 cloud_user@912cec8e983c:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1d9227acfe1c ubuntu "sleep 5000" About a minute ago Up About a minute hardcore_panini 705488122cfc ubuntu "sleep 5000" About a minute ago Up About a minute cranky_ardinghelli d24d261e2003 ubuntu "sleep 5000" About a minute ago Up About a minute keen_bhabha acd064dcd0ee ubuntu "sleep 5000" About a minute ago Up About a minute agitated_lovelace ac7eb755ec07 ubuntu "sleep 5000" About a minute ago Up About a minute compassionate_murdock 244812aab66a ubuntu "sleep 5000" About a minute ago Up About a minute goofy_haibt cloud_user@912cec8e983c:~$ docker system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 1 1 77.81MB 0B (0%) Containers 6 6 0B 0B Local Volumes 0 0 0B 0B Build Cache 0 0 0B 0B
cloud_user@912cec8e983c:~$ docker image pull ubuntu
Using default tag: latest
latest: Pulling from library/ubuntu
3153aa388d02: Pull complete
Digest:
sha256:0bced47fffa3361afa981854fcabcd4577cd43cebbb808cea2b1f33a3dd7f508
Status: Downloaded newer image for ubuntu:latest
cloud_user@912cec8e983c:~$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
ubuntu latest 5a81c4b8502e 4 weeks ago 77.8MB
cloud_user@912cec8e983c:~$ docker run --detach ubuntu sleep 5000
244812aab66aec47fe08455acf15757110e09a56fb6bc568af346ea84bb76397
cloud_user@912cec8e983c:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
244812aab66a ubuntu "sleep 5000" 4 seconds ago Up 3 seconds goofy_haibt
cloud_user@912cec8e983c:~$ docker run --detach ubuntu sleep 5000
ac7eb755ec0754446048e424f5b88b287d7f87f643b28cb84673cc4b917053c6
cloud_user@912cec8e983c:~$ docker run --detach ubuntu sleep 5000
acd064dcd0eee6e9867af764028dc4426c17c5adf988ea349e22ee3155f8f442
cloud_user@912cec8e983c:~$ docker run --detach ubuntu sleep 5000
d24d261e2003998b91a641cbc9aec71ecaa23fad048531368eadaa77319e215f
cloud_user@912cec8e983c:~$ docker run --detach ubuntu sleep 5000
705488122cfc5a43e4437b1384476a78bd9eadc25afb852d1ee6006db30324bb
cloud_user@912cec8e983c:~$ docker run --detach ubuntu sleep 5000
1d9227acfe1cea0debc48d494cff9937ea58007353f54a335801b54e9eae1e79
cloud_user@912cec8e983c:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1d9227acfe1c ubuntu "sleep 5000" About a minute ago Up About a minute hardcore_panini
705488122cfc ubuntu "sleep 5000" About a minute ago Up About a minute cranky_ardinghelli
d24d261e2003 ubuntu "sleep 5000" About a minute ago Up About a minute keen_bhabha
acd064dcd0ee ubuntu "sleep 5000" About a minute ago Up About a minute agitated_lovelace
ac7eb755ec07 ubuntu "sleep 5000" About a minute ago Up About a minute compassionate_murdock
244812aab66a ubuntu "sleep 5000" About a minute ago Up About a minute goofy_haibt
cloud_user@912cec8e983c:~$ docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 1 1 77.81MB 0B (0%)
Containers 6 6 0B 0B
Local Volumes 0 0 0B 0B
Build Cache 0 0 0B 0BContainers and the DoD
The DoD is a very different enterprise than most commercial businesses due to a wide range of applications, requirements, use cases and deployment environments coupled with the need to keep the barriers to entry for tools and applications low to support the warfighters and prevent the need for excessive training programs. It is clear to see how containers can help alleviate some of these problems but this is a large shift from traditional development, deployment and sustainment practices impacting the entire software acquisition lifecycle.
As mentioned earlier, containers do not fit every use case but they do fit well in many of them and can bring along with them all of the associated benefits. Companies like Amazon provide cloud hosted container running tools within GovCloud via Amazon Elastic Container Service (Amazon ECS) and even full blown orchestration environments like Kubernetes via Amazon Elastic Kubernetes Service (EKS). Meaning that once an application has been STIGed, containerized, scanned and approved, we can use managed services to host the application and apply operating system updates and maintenance.
On the other extreme, small edge hardware deployments can run containers right out of the box with Podman on Red Hat Enterprise Linux or any other Linux kernel down to a Raspberry Pi running an ARM based CPU. Similarly for a full blown container orchestrated environment Rancher's K3s distribution of Kubernetes provides a Kubernetes environment that can scale from a Raspberry Pi up to a cluster of many high powered servers.
Container hosting
There are a number of hosting options available within the DoD and most of them prefer the deployment of containers over other architecture. The reason for this goes back to accreditation and the preservation of their ATO. It is much ore manageable to scan and maintain a container that only includes the bare minimum set of software required to run versus an entire Virtual infrastructure of Operating systems, with independently managed security patches and requirements with their own ACAS/SCAP scans, etc going back to the pets and cattle concept. With the Container/cattle approach, you better communicate the risks of onboarding applications with regular (usually automated) code vulnerability and library scans on the container images themselves.
- Platform One
- CDAO Advana Edge
- Overmatch Software Armory (OSA)
- DISA's Hosting and Computer Center (HaCC)
Pipelines and DevOps
DevOps is central to the management many of the hosted platforms and enables them to provide the service while maintaining their own ATO. Ensuring that containerized applications are working as intended, secure and up to date can be automated in ways that traditional software cannot. This also means that the effective use of such tools can significantly reduce the time to deployment of application and their updates while significantly reducing cost.
The Rapid Asses and Incorporate Software Engineering (RAISE) process is a risk management framework specifically designed around containerized application delivery and DevSecOps practices that certifies environments as a RAISE Platform of choice (RPOC) and can reduce the ATO process from the traditional 18 months down to an average of 30-60 days for first time applications and as little as hors for updates! RAISE has proven so efficient that a Joint Memo for the Navy and Marine Corps now states that:
"Effective immediately, all programs with new software
starts and/or upgrades using containerized technologies must use
the RAISE 2.0 Implementation Guide."
Some of the platforms that are RPOCs or are in pursuit of becoming an RPOC are:
- Overmatch Software Factora (OSA)
- Virtual Secure Enclave (VSE)
- NAVAIR Arena
- BlackPerl
- USMC McBoss
- NAVSEA HM&E
Chapter 2: History and Components
History and Evolution of Containers
- 1970
chrootwas introduced to Unix 7- changes the root directory of a process and all child processes that are spun from it.
- Companies wanted to reduce rental fees of mainframes.
- Provides a limited view of the filesystem structure
- Adopted into Linux in 1982
- 1990s
- Linux Jail was created. This was a modified version of chroot is essentially the birth of virtualization and is still used in BSD today.
- The first concept of erecting better security between
chrootenvironments. - Adopted into FreeBSD as the
jailcommand - Provides its own users, network subsystems, and more.
- 2006
- Google engineers announced "process containers"
- Process containers isolate and limit resource usage of a group of process instead of a single process.
- These later became Control Groups/Cgroups to avoid confusing with the word container.
- Cgroups were later adopted into the Linux kernel which enabled the development of LXC containers.
- 2009
- UC Berkley RAD lab wanted to write software to better manage their computer cluster and leveraged Cgroups. this was originally called project Nexus but was renamed to Mesos due to a naming conflict.
- 2013
- Google open sourced their internal project LMCTFY (let me containerize that for you)
- A container stack that let containers manage their own sub-containers
- work on LMCTFY halted in 2015 when google donated their code to Docker and libcontainer
- 2013
- Docker released by Dotcloud as an open source project
- Enables us to package containers and move them from one area to another
- 2014
- Google open sources Kubernetes
- Originally based on their own internal container program called the "Borg system"
- Kubernetes was originally called Project Seven which is a Star Track reference and also why the kubernetes logo has seven spokes on the wheel. =)
Components of a Container
NOTE: The following content assumes a fundamental understanding of Linux permissions and directory structures. Outside of commands it is important to understand the difference between the root user (administrator) and the / directory often referred to as 'root'.
Chroot
- Changes the apparent root directory of a process.
- lets us create a limited view of our system
Chroot Exercise
An interactive lesson can be found on ACloudGuru here
Exercise summery
- Create a directory
- Create a few directories to hold libraries and programs
- bin
- lib64
- Add a group
- Add a user
cdinto the bin directory and copy over commands from the /bin directory- /ls
- /bash
- Copy over the libraries needed by the commands
ldd /bin/bashto list the libraries used by a command- copy the ones listed into your new lib64 directory
- do the same for
ldd /bin/ls
- Create some text file in the directory
- Run the chroot command
chroot /home/newdir/ /bin/bash - Try
lsand trycat - Try to cd up a directory
Linux Namespaces
- Takes a global resource and puts a wrapper around it make that resource appear more isolated.
- Currently 6 different Linux namespaces
- User - isolates security related identifiers and attributes things like UIDs and GIDs. This lets a process map a pid inside a container to a different pid inside the actual host.
- can be nested up to 32 times!
- This way even if a container gets compromised access to the root user (UID 1) they would not have elevated access on the host.
- IPC - Inter-process communication namespaces. Isolates systemd inetr-process communication
- creates a separate message queue for each namespace which allows containers to send and receive messages while still allowing isolation and security
- UTS - Unix timesharing namespace.
- Allows a single system to appear to have a different host and domain name.
- Lets containers have their own names ensuring applications can use host names as defualts when communicating with containers
- Ensures we can still use traditional commands like
unameandhostname
- Mount - controls the file system mount points that are visible to each container
- lets us provide an isolated list of mounted filesystems
- PID
- allows the isolation of process ID numbers
- spins a new process tree for every container
- lets us supend and move processes to a new host without conflict
- Network
- lets a container have its own network stack
- routing table
- firewall rules
- network devices
- lets a container have its own network stack
- User - isolates security related identifiers and attributes things like UIDs and GIDs. This lets a process map a pid inside a container to a different pid inside the actual host.
- Some think that cgroups would be the same but cgroups limit access whereas namespaces don't even let you "See" the resource.
Network Namespaces Exercise
The following steps are from this ACloudGuru Exercise
If following along independently, be sure to create an Ubuntu machine for the below exercise:
## create a new namespace named sample1 sudo ip netns add sample1 sudo ip netns list ## list host iptables rules iptables -L # Note this does not work without sudo sudo iptables -L # execute a command within our new namespace sudo ip netns exec sample1 iptables -L sudo ip netns exec sample1 bash # starts a bash shell within that network namespace ## this will change you to the root user within the namespace # add a new rule iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT # List the new addition iptables -L exit #exit the namespace # List host system rules sudo iptables -L
## create a new namespace named sample1
sudo ip netns add sample1
sudo ip netns list
## list host iptables rules
iptables -L # Note this does not work without sudo
sudo iptables -L
# execute a command within our new namespace
sudo ip netns exec sample1 iptables -L
sudo ip netns exec sample1 bash # starts a bash shell within that network namespace
## this will change you to the root user within the namespace
# add a new rule
iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
# List the new addition
iptables -L
exit #exit the namespace
# List host system rules
sudo iptables -LControl Groups
- Cgroups are a Linux kernel feature that limits, accounts, and isolates the resource usage (CPU, memory, disk I/O, network, etc.) of a collection of processes.
- cgroups do NOT limit what a process can see
- There are multiple versions of cgroups v1 and v2 both of which are still in use today.
- In v1 things got messy due to too many developers/companies doing things differently
- v2 was created to standardize but not fully adopted as a complete replacement
- Today v1 is still found in a number of legacy systems
- Cgroups allow each subsystem to have it own process hierarchy. Cgroups hierarchy is allowed to be independent from another allowing one process to live in multiple trees!
Common cgroup subsystems:
- blkio - limits and measures the amount of I/O for each group of processes letting you throttle limits
- cpu - allows you to monitor CPU usage and set weights to track usage per CPU
- cpuacct - generates automatic reports on CPU resources used by tasks in a cgroup
- cpuset - allows you to pin groups of processes or groups of a process to one CPU
- devices - lets you deny access to a device by tasks in a cgroup.
- freezer - allows you do a
sigstopto an entire container - memory - sets limits on memory used by tasks in a cgroup and generates reports of memory resources
- net_cls - tags network packets with a
classidthat allows the identification of packets originating from a particular cgroup task. This means that you could configure package control on your network for quality of service like functionality - net_prio - provides a way to set the priority of network traffic dynamically
SELinux
-
On RHEL / OpenShift instances (as well as some others) SELinux also provides security
-
SELinux is a deny-first policy within the Linux kernel that strictly identifies what files, ports, etc a process can interact with
-
Containers don't conform well to the ideas of SELinux and so instead there are contexts developed to generically identify that a resource can be used by containers.
-
NOTE: SELinux policies also don't translate inside of a container which is a topic currently being worked within the DoD
Seccomp policies
limit what kernel arguments a container can execute.
- Docker has a default policy that is applied but can be modified
- Tools like "Confine" have been funded by ONR to help create tools that automate the process of creating SecCcomp filters by scanning what it actually is using.
- Prevents unintended behavior of a running container.
Chapter 3: Virtualization and LXC
Virtual machines
- A host operating system will run a hypervisor
- Type 2 - Software based hypervisor like VirtualBox
- Type 1 - Directly exposes hardware to the VM to improve performance. There is still an operating system but it commonly is not used for anything other than virtualization (think ESXi)
Characteristics of a hypervisor:
- Hypervisors provide virtual hardware to the guest operating system
- Guest Machines require a full fledged setup process just like a physical system would.
- Increases the density of a server but also the threat space and amount of maintenance required.
Linux Containers (LXC)
Linux Containers. These are effectively a middle ground between a containerized environment and a full VM.
LXC Containers use some of the same concepts that we already discussed.
- chroot
- Kernel Namespaces
- SELinux/Apparmor
- Seccomp policies
LXC lets you create an isolated environment quickly by sharing the Linux Kernel. The linux distribution no longer matters however, LXC can only create linux based environments and does not support operating systems outside of the Linux Kernel.
System administrators that are familiar with virtualization often find LXC containers to be easy to understand, manage and work with however, they are not the predominant form of containerization in use today.
LXC Use Case
A developer wrote a python application in python2 on his/her Ubuntu machine and wants to test it out on RHEL but doesn't have a RHEL system. Or even wants to make sure that the application works on python 3 but is worried about their developer machine getting messed up. You could do this with VMs but that could take more time to set up than to actually test. Not to mention the hardware requirements to support it all... Instead, use a CONTAINER!
LXC/LXD
LXD is the linux daemon (hence the D) that is lets you interface with LXC applications (little confusing).
- Developed at IBM
- Mostly a focus of Debian/Ubuntu based distributions (installed as a snap package) and is actually now a part of Canonical (the makers of Ubuntu.)
- The one underlying technology behind Proxmox
- Also what is used on ChromeOS to let you run Linux commands! You can even access and use it on your own Chromebook
Command Line example
This example should be done on an Ubuntu based operating systems
sudo snap install lxd # This may report that lxd is already installed by default sudo lxd init # Go through the setup by just accepting the defaults sudo lxc list # Lists all the lxc containers on this system sudo lxc launch ubuntu:22.04 # will pull the image from the repository and run a container with a randomly generated name sudo lxc launch ubuntu:22.04 my-ubuntu # launches another container with a the name 'my-ubuntu' sudo lxc list sudo lxc launch images:alpine/3.18 my-alpine # pulls from a remote server # Enter the new alpine container and create a simple text file sudo lxc exec my-alpine -- /bin/ash cat /etc/os-release echo hello > hi.txt ls exit # Now enter the previous Ubuntu container and see if the text file is available sudo lxc exec my-ubuntu -- /bin/bash cat /etc/os-release ls exit
sudo snap install lxd # This may report that lxd is already installed by default
sudo lxd init # Go through the setup by just accepting the defaults
sudo lxc list # Lists all the lxc containers on this system
sudo lxc launch ubuntu:22.04 # will pull the image from the repository and run a container with a randomly generated name
sudo lxc launch ubuntu:22.04 my-ubuntu # launches another container with a the name 'my-ubuntu'
sudo lxc list
sudo lxc launch images:alpine/3.18 my-alpine # pulls from a remote server
# Enter the new alpine container and create a simple text file
sudo lxc exec my-alpine -- /bin/ash
cat /etc/os-release
echo hello > hi.txt
ls
exit
# Now enter the previous Ubuntu container and see if the text file is available
sudo lxc exec my-ubuntu -- /bin/bash
cat /etc/os-release
ls
exit
Remotes
A system that your lxc environment is connected to is considered a "remote." Remotes can supply a range of functions such as providing operating system images for you to use or they can also be host machines on your local network where the lxc container is running.
You can list all of the remotes on your system with the below command:
lxc remote list
lxc remote listit is considered a best practice to only pull official images from https://images.linuxcontainers.org
Chapter 4: Understanding Docker
Docker started as a python script for Dotcloud in 2008 but by 2012 it had grown to over 100 microservices. When Docker was open sourced it was so well received that Dotcloud ended up changing its company name to docker inc!
What is Docker
An application that makes it easier for us to create, deploy and run other applications inside linux containers. it actually started as a wrapper around LXC. It brings a powerful set of capabilities and functions:
- Portable Deployment Across Machine
- Application-Centric Design
- Automated Build
- Version controlling
- Component Re-Use
- Image Sharing
- Large Tool Echosystem
Container Runtimes
Today Docker is still perhaps the most well known container tool and continues to provide developers and users with a straightforward interface for the development and running of containers However, in 2021 Docker made the Docker Desktop tool a paid application for large companies (Docker Engine) and while Docker Engine continues to be free it is not well advertised and documented as compared to the Desktop application requiring MacOS users to install via homebrew and Windows users to start recommending WSL! For this reason among others, the competition space of container runtimes has grown significantly in recent years resulting in many container Runtimes including podman, cri-o, and the isolation of containerd. Although there are many options, they all run the same type of containers under the Open Container Initiative meaning that you can develop an application with Docker and run it with Podman, rkt or any other compliant runtime!
Getting started
Docker runs as a long running process called a daemon on your machine and exposes a socket and API that lets you communicate with the docker daemon. The docker CLI lets you run commands that send the API calls for you.
Installing Docker Engine
This exercise installs Docker on a RHEL 8 machine. Similar commands for installing in other environments are available in the docker documentation (NOTE: that only "Docker Server" is technically free for corporate use).
- Install prerequisites
sudo yum install -y yum-utils lvm2 device-mapper-persistent-data
sudo yum install -y yum-utils lvm2 device-mapper-persistent-data - Configure the Docker repository
sudo yum install -y yum-utils sudo yum-config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo
sudo yum install -y yum-utils sudo yum-config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo - Install the
docker-cepackagesudo dnf install docker-ce # hit y for 'yes' notice this also installs the container-selinux package # confirm the GPG fingerprint
sudo dnf install docker-ce # hit y for 'yes' notice this also installs the container-selinux package # confirm the GPG fingerprint - start and enable the daemon
sudo systemctl enable docker sudo systemctl start docker
sudo systemctl enable docker sudo systemctl start docker - Add users who will be using Docker to the
dockergroup so they can use it without sudo privilegessudo usermod -aG docker user # log out and back into the system to make sure the new group is effective
sudo usermod -aG docker user # log out and back into the system to make sure the new group is effective
Running a basic container
- Run the
hello-worldcontainer with theruncommanddocker run hello-world
docker run hello-world - list running containers with the
pscommanddocker ps
docker ps - list all running and stopped containers with
ps -adocker ps -a
docker ps -a
Container images
Docker images are made with copy on write (COW) technology which means whenever we want to make a change, we first must copy the file and make a change there.
As a result Images are made of multiple layers:
- Base layer - A starting point. Typically this is an operating system or application framework
- Layers - A copy of the previous layer with whatever change you need applied to it.
Container Registry
A container registry is a place (on the internet or internal network) to store container images. Not to be confused with a source code repository, a registry may or may not have code that you can see or that is safe to use. It is considered a best practice to pull container images a signed and trusted registry or at a minimum when pulling from hub.docker.com, you should be sure to look for the "DOCKER OFFICIAL IMAGE" or "VERIFIED PUBLISHER" label.
Some of the example container registries are:
- docker.io - Docker's official container registry where anybody with an account can push an image. Be careful which images you chose!
- https://hub.docker.com/ for searching
- registry.redhat.io - Red Hat's registry where they verify and sign every image available.
- See https://catalog.redhat.com/software/containers/search for searching
- registry1.dso.mil - Platform One's Iron Bank harbor registry. A USAF CAC authenticated site where containers are hardened, hosted and continually scanned for use within the DoD.
- See https://ironbank.dso.mil/repomap for searching
There are many others hosted by various cloud providers, as part of GitLab/SPORK or even internally hosted as a container itself!
Using an image
We can use common docker commands to pull and inspect images:
- Pull the latest version of the alpine image.
docker image pull alpine:latest
docker image pull alpine:latest - Take a look at the history of that newly pulled image
docker history alpine
docker history alpine - List all of the images that are locally stored on this machine
docker images
docker images - Pull a slightly larger apache (httpd) container
docker image pull httpd # it will default to the :latest tag # This one has more layers in it than the last container docker images
docker image pull httpd # it will default to the :latest tag # This one has more layers in it than the last container docker images - Take a look at the history of the
httpdcontainer to see all the layers that were used to build the imagedocker history httpd --no-trunc # hard to read
docker history httpd --no-trunc # hard to read - Go over to hub.docker.com and check out the dockerfile itself
- Go to https://hub.docker.com
- Search or httpd and find the 'official' repository (usually at the top)
- Check out the Dockerfile and compare it to the
docker historyoutput
Image layer sharing
Container images often share layers between them and it makes no sense to duplicate them across containers waisting hard disk space, so instead container images only store the layers that diverge or are not shared with any other container images
- Find the apache Dockerfile from hub.docker.com
- look for the source image and pull it the image
docker image pull debian:bullseye-slim
docker image pull debian:bullseye-slim - Observe how the image layer sharing reduces disk usage via the
system df -vcommanddocker system df -v
docker system df -v
The Dockerfile
While there are a large number of container images available, you will inevitably need to create your own. Perhaps this is to host a new application, or to add some tweaks/modifications to an existing one to make it more secure (IronBank images are exactly this).
The Dockerfile is the most well known way to define a new image. It is however, not the only tool that uses this format. For this reason you may see these files called Containerfiles instead of Dockerfiles.
Example
A Dockerfile/Containerfile is nothing more than a text file containing the steps for creating a container image. These files always begin with the 'base image' that you will start from followed by a number of steps to perform within it. This could be installing new packages, configuring tools, removing unused applications, closing vulnerabilities, etc.
- Create a new directory that will hold our applicaiton
mkdir myapp cd myapp
mkdir myapp cd myapp - Create a text file named
Dockerfileusing your editor of choice and include the followingFROM ubuntu:22.04 LABEL maintainer="david.d.developer@us.navy.mil" RUN apt update RUN apt install -y python3
FROM ubuntu:22.04 LABEL maintainer="david.d.developer@us.navy.mil" RUN apt update RUN apt install -y python3 - Create a new container image!
docker build . ... output omitted cloud_user@912cec8e982c:~/foo$ docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE <none> <none> 1cc8e77160e0 5 seconds ago 147MB
docker build . ... output omitted cloud_user@912cec8e982c:~/foo$ docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE <none> <none> 1cc8e77160e0 5 seconds ago 147MB - You most likely want to give the image a name, so rerun the build using the
-tflagdocker build -t mypython:0.1 . docker image ls
docker build -t mypython:0.1 . docker image ls - lets try out our new container!
# Run a python3 version check on the container docker run --rm mypython:0.1 python3 -V # Now to compare to the original ubuntu image docker run --rm ubuntu:22.04 python3 -V
# Run a python3 version check on the container docker run --rm mypython:0.1 python3 -V # Now to compare to the original ubuntu image docker run --rm ubuntu:22.04 python3 -V
Dockerfile References
- The Dockerfile is not the only way to create a new image. Containers can be defined non-declaratively (not recommended for security reasons) or using other workflow tools like buildah
- For details about the Dockerfile and the various commands within it see https://docs.docker.com/engine/reference/builder/
Running Containers
Now that we've built the containers, it would be nice be able to run them. Docker provides a lot of commands but in this class we will highlight some of the most commonly used.
Understanding commands
Each command in docker has a very convenient help command that can be extremely useful when learning.
cloud_user@912cec8e982c:~$ docker --help Usage: docker [OPTIONS] COMMAND A self-sufficient runtime for containers Common Commands: run Create and run a new container from an image exec Execute a command in a running container ps List containers build Build an image from a Dockerfile pull Download an image from a registry ... additional output omitted
cloud_user@912cec8e982c:~$ docker --help
Usage: docker [OPTIONS] COMMAND
A self-sufficient runtime for containers
Common Commands:
run Create and run a new container from an image
exec Execute a command in a running container
ps List containers
build Build an image from a Dockerfile
pull Download an image from a registry
... additional output omitted To Get help on a specific sub-command simply add it to the command line argument int the form of docker <command> --help
cloud_user@912cec8e982c:~$ docker run --help Usage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...] Create and run a new container from an image Aliases: docker container run, docker run Options: ... additional output omitted
cloud_user@912cec8e982c:~$ docker run --help
Usage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...]
Create and run a new container from an image
Aliases:
docker container run, docker run
Options:
... additional output omitted Warning: while you can sometimes omit the -- characters from --help in some commands the word help without -- may be seen as an argument and interpreted as such.
cloud_user@912cec8e982c:~$ docker run help Unable to find image 'help:latest' locally docker: Error response from daemon: pull access denied for help, repository does not exist or may require 'docker login': denied: requested access to the resource is denied. See 'docker run --help'.
cloud_user@912cec8e982c:~$ docker run help
Unable to find image 'help:latest' locally
docker: Error response from daemon: pull access denied for help, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.Tab completion also works on most of these commands, where you can simply press the <tab> key on your keyboard twice.
cloud_user@912cec8e982c:~$ docker run <tab><tab> debian hello-world httpd mariadb simplicitie/katacoda-hellogo debian:bullseye-slim hello-world:latest httpd:latest mariadb:10.11.3 simplicitie/katacoda-hellogo:latest cloud_user@912cec8e982c:~$ docker run -<tab><tab> Display all 111 possibilities? (y or n) --add-host --cpuset-mems --env-file --kernel-memory --no-healthcheck --shm-size -P --attach --detach --expose --label --oom-kill-disable --sig-proxy=false -a --blkio-weight --detach-keys --gpus --label-file --oom-score-adj --stop-signal -c --blkio-weight-device --device --group-add --link --pid --stop-timeout -d --cap-add --device-cgroup-rule --health-cmd --link-local-ip --pids-limit --storage-opt -e --cap-drop --device-read-bps --health-interval --log-driver --platform --sysctl -h --cgroup-parent --device-read-iops --health-retries --log-opt --privileged --tmpfs -i --cgroupns --device-write-bps --health-start-period --mac-address --publish --tty -l --cidfile --device-write-iops --health-timeout --memory --publish-all --ulimit -m --cpu-period --disable-content-trust=false --help --memory-reservation --pull --user -p --cpu-quota --dns --hostname --memory-swap --quiet --userns -q --cpu-rt-period --dns-option --init --memory-swappiness --read-only --uts -t --cpu-rt-runtime --dns-search --interactive --mount --restart --volume -u --cpu-shares --domainname --ip --name --rm --volume-driver -v --cpus --entrypoint --ip6 --network --runtime --volumes-from -w --cpuset-cpus --env --ipc --network-alias --security-opt --workdir
cloud_user@912cec8e982c:~$ docker run <tab><tab>
debian hello-world httpd mariadb simplicitie/katacoda-hellogo
debian:bullseye-slim hello-world:latest httpd:latest mariadb:10.11.3 simplicitie/katacoda-hellogo:latest
cloud_user@912cec8e982c:~$ docker run -<tab><tab>
Display all 111 possibilities? (y or n)
--add-host --cpuset-mems --env-file --kernel-memory --no-healthcheck --shm-size -P
--attach --detach --expose --label --oom-kill-disable --sig-proxy=false -a
--blkio-weight --detach-keys --gpus --label-file --oom-score-adj --stop-signal -c
--blkio-weight-device --device --group-add --link --pid --stop-timeout -d
--cap-add --device-cgroup-rule --health-cmd --link-local-ip --pids-limit --storage-opt -e
--cap-drop --device-read-bps --health-interval --log-driver --platform --sysctl -h
--cgroup-parent --device-read-iops --health-retries --log-opt --privileged --tmpfs -i
--cgroupns --device-write-bps --health-start-period --mac-address --publish --tty -l
--cidfile --device-write-iops --health-timeout --memory --publish-all --ulimit -m
--cpu-period --disable-content-trust=false --help --memory-reservation --pull --user -p
--cpu-quota --dns --hostname --memory-swap --quiet --userns -q
--cpu-rt-period --dns-option --init --memory-swappiness --read-only --uts -t
--cpu-rt-runtime --dns-search --interactive --mount --restart --volume -u
--cpu-shares --domainname --ip --name --rm --volume-driver -v
--cpus --entrypoint --ip6 --network --runtime --volumes-from -w
--cpuset-cpus --env --ipc --network-alias --security-opt --workdir Using the run command
You can run a container fairly easily by specifying docker run <container_image_name>. If the container image is not already available locally, the run command will automatically search for it and pull the image down from https://hub.docker.com (by default) and then run.
cloud_user@912cec8e982c:~$ docker run hello-world
cloud_user@912cec8e982c:~$ docker run hello-worldThe hello-wrold container simply displays text and then exits but the container itself is still on your system so that it could be restarted.
cloud_user@912cec8e982c:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES cloud_user@912cec8e982c:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5a4a38a2f0f2 hello-world "/hello" 6 minutes ago Exited (0) 6 minutes ago sleepy_vaughan
cloud_user@912cec8e982c:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cloud_user@912cec8e982c:~$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5a4a38a2f0f2 hello-world "/hello" 6 minutes ago Exited (0) 6 minutes ago sleepy_vaughanThe docker ps command will display all the containers that are currently running on the host. to display all containers on the host including stoped and created containers, you can pas the -a flag.
Your container will be named something different. Unless specified in the run commands with the --name argument, Docker will generate a name.
To remove the stopped container use the docker rm command
cloud_user@912cec8e982c:~$ docker container rm sleepy_vaughan sleepy_vaughan cloud_user@912cec8e982c:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cloud_user@912cec8e982c:~$ docker container rm sleepy_vaughan
sleepy_vaughan
cloud_user@912cec8e982c:~$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESLet's run a container that runs a service constantly instead of doing a small job and exiting. For this we will use the apache web service known as "httpd". and this time lets specify the name with the --name flag
docker run --name webserver httpd ... output omitted
docker run --name webserver httpd
... output omittedYou will notice that you can no longer type commands thats because the container is currently running in the foreground. use <ctrl>+c to close it which will actually stop the container. We can see that the container is still present our our system and actually restart it as well.
cloud_user@912cec8e982c:~$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f5e22a69ce4e httpd "httpd-foreground" 2 minutes ago Exited (0) 59 seconds ago webserver cloud_user@912cec8e982c:~$ docker start webserver webserver cloud_user@912cec8e982c:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f5e22a69ce4e httpd "httpd-foreground" 3 minutes ago Up 22 seconds 80/tcp webserver
cloud_user@912cec8e982c:~$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f5e22a69ce4e httpd "httpd-foreground" 2 minutes ago Exited (0) 59 seconds ago webserver
cloud_user@912cec8e982c:~$ docker start webserver
webserver
cloud_user@912cec8e982c:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f5e22a69ce4e httpd "httpd-foreground" 3 minutes ago Up 22 seconds 80/tcp webserverYou will notice that the container is now running in the background. We can stop and clean it up with when we are done.
cloud_user@912cec8e982c:~$ docker stop webserver webserver cloud_user@912cec8e982c:~$ docker rm webserver webserver
cloud_user@912cec8e982c:~$ docker stop webserver
webserver
cloud_user@912cec8e982c:~$ docker rm webserver
webserverSome common docker run flags
--name stringAssign a name to the container-d,--detachRun container in background and print container ID-e list,--env listSet environment variables--rmAutomatically remove the container when it exits-i,--interactiveKeep STDIN open even if not attached-t,--ttyAllocate a pseudo-TTY-itcommonly used combination of-iand-tto access Interactive terminal mode-v list,--volume listBind mount a volume-p list,--publish listPublish a container's port(s) to the host
Chapter 5: Advanced Docker
Environment variables
Environment variables are commonly used to configure or modify a container's default settings.
For example lets say you want to use the popular MariaDB database but instead of figuring out how to install and configure it, why not use a container!
You go to run the container just like any other
cloud_user@912cec8e982c:~$ docker run mariadb:10.11.3
2023-05-12 13:35:51+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.3+maria~ubu2204 started.
2023-05-12 13:35:51+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2023-05-12 13:35:51+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.3+maria~ubu2204 started.
2023-05-12 13:35:52+00:00 [ERROR] [Entrypoint]: Database is uninitialized and password option is not specified
You need to specify one of MARIADB_ROOT_PASSWORD, MARIADB_ROOT_PASSWORD_HASH, MARIADB_ALLOW_EMPTY_ROOT_PASSWORD and MARIADB_RANDOM_ROOT_PASSWORD
cloud_user@912cec8e982c:~$ docker run --detach --name some-mariadb --env MARIADB_USER=example-user --env MARIADB_PASSWORD=my_cool_secret --env MARIADB_ROOT_PASSWORD=my-secret-pw mariadb:10.11.3 cloud_user@912cec8e982c:~$ docker run mariadb:10.11.3
2023-05-12 13:35:51+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.3+maria~ubu2204 started.
2023-05-12 13:35:51+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2023-05-12 13:35:51+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.3+maria~ubu2204 started.
2023-05-12 13:35:52+00:00 [ERROR] [Entrypoint]: Database is uninitialized and password option is not specified
You need to specify one of MARIADB_ROOT_PASSWORD, MARIADB_ROOT_PASSWORD_HASH, MARIADB_ALLOW_EMPTY_ROOT_PASSWORD and MARIADB_RANDOM_ROOT_PASSWORD
cloud_user@912cec8e982c:~$ docker run --detach --name some-mariadb --env MARIADB_USER=example-user --env MARIADB_PASSWORD=my_cool_secret --env MARIADB_ROOT_PASSWORD=my-secret-pw mariadb:10.11.3 This makes sense because it would be very insecure to have configurations like passwords set by default where they could be easily overlooked.
Check the hub.docker.com website to see the documentation
https://hub.docker.com/_/mariadb
specifically note the "How to use this image" section
docker run --detach --name some-mariadb --env MARIADB_USER=example-user --env MARIADB_PASSWORD=my_cool_secret --env MARIADB_ROOT_PASSWORD=my-secret-pw mariadb:10.11.3
docker run --detach --name some-mariadb --env MARIADB_USER=example-user --env MARIADB_PASSWORD=my_cool_secret --env MARIADB_ROOT_PASSWORD=my-secret-pw mariadb:10.11.3lets take a closer look at some of those arguments:
--detachmakes sure it so that the container runs in the background and doesn't consume our terminal--namemakes it easier to identify the container. If not provided, docker will randomly generate a sometimes funny name.--envis how we set environment variables byVARIABLENAME=value- Most variable names are capitalized as a best practice but it is not strictly necessary.
- This flag can be used multiple times to send multiple environment variables
Environment variables are common within Unix/Linux operating systems and are an essential part of how the system works. You can actually see them within the container:
- Verify the container is running
cloud_user@912cec8e982c:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES e1557ee91e9b mariadb:10.11.3 "docker-entrypoint.s…" 4 minutes ago Up 4 minutes 3306/tcp some-mariadb
cloud_user@912cec8e982c:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e1557ee91e9b mariadb:10.11.3 "docker-entrypoint.s…" 4 minutes ago Up 4 minutes 3306/tcp some-mariadb- Enter the container interactively
cloud_user@912cec8e982c:~$ docker exec -it some-mariadb /bin/bash
cloud_user@912cec8e982c:~$ docker exec -it some-mariadb /bin/bash- Retrieve the environment variables via command line functions
root@e1557ee91e9b:/# echo $MARIADB_USER example-user root@e1557ee91e9b:/# echo $MARIADB_PASSWORD my_cool_secret root@e1557ee91e9b:/# env
root@e1557ee91e9b:/# echo $MARIADB_USER
example-user
root@e1557ee91e9b:/# echo $MARIADB_PASSWORD
my_cool_secret
root@e1557ee91e9b:/# envPort Mapping
Unfortunately hosting a database as a container does little to no good if you cannot connect to it and by default Docker containers are not exposed to external networks. Docker provides a number of ways in which we can 'expose' a container both inside and outside of our host as well as other networking features. For now lest look at port mapping.
- Run the following container which hosts a web server on port
3000cloud_user@912cec8e982c:~$ docker run -d --rm --name web simplicitie/katacoda-hellogo cb1c958521cb6bb1342bce2ba7e6f441a725876fddd852565b8e76f3c47bfa2e cloud_user@912cec8e982c:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES cb1c958521cb simplicitie/katacoda-hellogo "/hellogo" 17 seconds ago Up 16 seconds web
cloud_user@912cec8e982c:~$ docker run -d --rm --name web simplicitie/katacoda-hellogo cb1c958521cb6bb1342bce2ba7e6f441a725876fddd852565b8e76f3c47bfa2e cloud_user@912cec8e982c:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES cb1c958521cb simplicitie/katacoda-hellogo "/hellogo" 17 seconds ago Up 16 seconds web - obtain the IP address of the container. The
docker inspectcommands gives a lot of information but we can filter out just the IP address using the below commandcloud_user@912cec8e982c:~$ docker container inspect -f '{{ .NetworkSettings.IPAddress }}' web 172.17.0.3cloud_user@912cec8e982c:~$ docker container inspect -f '{{ .NetworkSettings.IPAddress }}' web 172.17.0.3 - The container was built to host the web server on port
3000so lets see ensure we can access that by using the curl commandcloud_user@912cec8e982c:~$ curl 172.17.0.3:3000 Hello, World!cloud_user@912cec8e982c:~$
cloud_user@912cec8e982c:~$ curl 172.17.0.3:3000 Hello, World!cloud_user@912cec8e982c:~$ - Great it works! sadly this is only available internally on this machine unless we map the port to a port on the host machine. We can use any port but for this example we will use the commonly used port
8080which we can map to the container using the-p host_port:container_portflag.cloud_user@912cec8e982c:~$ docker stop web web # Notice we do not need to delete the container becuase we started it with the --rm argument cloud_user@912cec8e982c:~$ docker run -p 8080:3000 -d --rm --name web simplicitie/katacoda-hellogo 2ce41040aa68f5c73dc24bc713184190581b3a5e9edd9d31555bf6c931962b2e cloud_user@912cec8e982c:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2ce41040aa68 simplicitie/katacoda-hellogo "/hellogo" 2 seconds ago Up 1 second 0.0.0.0:8080->3000/tcp, :::8080->3000/tcp web
cloud_user@912cec8e982c:~$ docker stop web web # Notice we do not need to delete the container becuase we started it with the --rm argument cloud_user@912cec8e982c:~$ docker run -p 8080:3000 -d --rm --name web simplicitie/katacoda-hellogo 2ce41040aa68f5c73dc24bc713184190581b3a5e9edd9d31555bf6c931962b2e cloud_user@912cec8e982c:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2ce41040aa68 simplicitie/katacoda-hellogo "/hellogo" 2 seconds ago Up 1 second 0.0.0.0:8080->3000/tcp, :::8080->3000/tcp web - we can now curl the system locally or even access the container via a web browser. NOTE: you may need to configure firewall rules to allow port 8080 to the host machine.
cloud_user@912cec8e982c:~$ curl localhost:8080 Hello, World!cloud_user@912cec8e982c:~$ cloud_user@912cec8e982c:~$ curl 912cec8e982c.mylabserver.com:8080 Hello, World!cloud_user@912cec8e982c:~$
cloud_user@912cec8e982c:~$ curl localhost:8080 Hello, World!cloud_user@912cec8e982c:~$ cloud_user@912cec8e982c:~$ curl 912cec8e982c.mylabserver.com:8080 Hello, World!cloud_user@912cec8e982c:~$
Practice Lab
Persistant Storage
So far we've seen containers that don't really store anything. The MariaDB server we stood up earlier was completely empty but still was able to store some information. Container persistance is an important topic for some applications but thankfully it is fairly straightforward via what are called volumes.
Local filesystem mount points
One of the simplest ways to preserve your data is by mounting a directory within your host system to the container. This way the container and host OS both see the same folder and data and can work with it interchangeably. Users should be careful with this feature as you may inadvertently expose more to the application than what is necessary but it can be quite useful.
In the below example, a Red Hat UBI8 container runs and downloads rpm files required for the bzip2 package locally onto the host's present working directory (pwd) via a volume mount.
mkdir rpms cd rpms podman run --rm -it -v $(pwd):/dest redhat/ubi8 dnf install -y --downloadonly --destdir=/dest bzip2
mkdir rpms
cd rpms
podman run --rm -it -v $(pwd):/dest redhat/ubi8 dnf install -y --downloadonly --destdir=/dest bzip2Anonymous volumes
A container like a PostgreSQL or MariaDB database uses the VOLUME command within it's image build and will create an anonymous container volumes with a random string of hex characters for their names (unless a named volume is specified) so that the containers can start and stop without losing their data.
When a container is removed, any anonymous volumes attached to it will be cleaned up automatically by docker. This does not apply to containers that are simply Exited but does automatically apply when a container that was created with the --rm flag is finished.
Named volumes
Named docker volumes are technically no different than anonymous volumes but have a much easier to identify name associated with them. It is important to note than a named volume will NOT automatically be removed when the containers that use it are deleted allowing for it to be moved from one container to another.
docker volume create website
docker volume create websiteWhen running a container that uses a named volume, you simply provide the volume name to the -v flag. The below example (from the Exercise) specifically mounts it as read only which is a handy security feature.
docker run -d --name web1 -p 80:80 -v website:/usr/local/apache2/htdocs:ro httpd
docker run -d --name web1 -p 80:80 -v website:/usr/local/apache2/htdocs:ro httpdVolume location on host
Docker volumes are simply directories within your local filesystem and can be easily searched although they do have Posix standard group, owner, read write and execute permission settings which may require you to sudo.
You can inspect where a volume is mounted as well as where it is located locally via the command:
docker inspect db1 -f '{{ json .Mounts}}' | python -m json.tooldocker inspect db1 -f '{{ json .Mounts}}' | python -m json.toolVolumes are stored in /var/lib/docker/volumes/<volume_name>/_data by default where the volume name will be a random set of hex characters for Anonymous volumes.
Exercise
Storing Container Data In Docker Volumes
Docker Networking
Previously we created a MariaDB container with the below command:
docker run --detach --name some-mariadb --env MARIADB_USER=example-user --env MARIADB_PASSWORD=my_cool_secret --env MARIADB_ROOT_PASSWORD=my-secret-pw mariadb:10.11.3
docker run --detach --name some-mariadb --env MARIADB_USER=example-user --env MARIADB_PASSWORD=my_cool_secret --env MARIADB_ROOT_PASSWORD=my-secret-pw mariadb:10.11.3Locally connecting
Through the docker ps command we can see that the container exposes port 3306 and we can actually access that from the host machine but first we need to know what internal IP that container is hosted on. we can get additional container details from the docker inspect command
cloud_user@912cec8e982c:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e2b5ef152b0c mariadb:10.11.3 "docker-entrypoint.s…" 3 minutes ago Up 3 minutes 3306/tcp some-mariadb
cloud_user@912cec8e982c:~$ docker inspect some-mariadb | grep -i IPAddress
"SecondaryIPAddresses": null,
"IPAddress": "172.17.0.2",
"IPAddress": "172.17.0.2",cloud_user@912cec8e982c:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e2b5ef152b0c mariadb:10.11.3 "docker-entrypoint.s…" 3 minutes ago Up 3 minutes 3306/tcp some-mariadb
cloud_user@912cec8e982c:~$ docker inspect some-mariadb | grep -i IPAddress
"SecondaryIPAddresses": null,
"IPAddress": "172.17.0.2",
"IPAddress": "172.17.0.2",In order to connect to it we can use another mariadb container remember that we used an environment variable to set the password to "my_cool_secret" on the original server container.
cloud_user@912cec8e982c:~$ docker run -it --rm mariadb:10.11.3 mariadb -h 172.17.0.2 -u example-user -p Enter password: Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 3 Server version: 10.11.3-MariaDB-1:10.11.3+maria~ubu2204 mariadb.org binary distribution Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> exit Bye
cloud_user@912cec8e982c:~$ docker run -it --rm mariadb:10.11.3 mariadb -h 172.17.0.2 -u example-user -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 3
Server version: 10.11.3-MariaDB-1:10.11.3+maria~ubu2204 mariadb.org binary distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> exit
ByeThis works okay for accessing it on this particular machine but what if we wanted to access it externally? We need to know the IP address or hostname or port of your machine which can be found on the "Cloud Playground" page. Lets use that to try to access the database:
NOTE: Our training environment does have some additional limitations on what ports are routed to/from your VM so in this case we must use the "Private IPv4" address however, this restriction does not apply outside of this cloud hosted environment.
Externally connecting
# You will need to replace the -h <ip_address> with the Private IPv4 of your host machine docker run -it --rm mariadb:10.11.3 mariadb -h 172.31.122.13 -u example-user -p Enter password: ERROR 2002 (HY000): Can't connect to server on '172.31.122.13' (115)
# You will need to replace the -h <ip_address> with the Private IPv4 of your host machine
docker run -it --rm mariadb:10.11.3 mariadb -h 172.31.122.13 -u example-user -p
Enter password:
ERROR 2002 (HY000): Can't connect to server on '172.31.122.13' (115)We can use port mapping to fix this by mapping a host port into a port on the container. The host port does not need to be the same either which lets us easily create multiple instances of a container without worrying about them using the same port on the host! to accomplish this we use the -p flag when we run the container with the syntax of -p host_port:container_port for now we will keep them the same. You will also need to expose the port through any firewalls on your host system.
docker stop some-mariadb docker rm some-mariadb docker run --detach --name some-mariadb -p 3306:3306 --env MARIADB_USER=example-user --env MARIADB_PASSWORD=my_cool_secret --env MARIADB_ROOT_PASSWORD=my-secret-pw mariadb:10.11.3 # we also need to open the port on our firewall sudo ufw allow 3306 Rule added Rule added (v6) # You will need to replace the -h <ip_address> with the Private IPv4 of your host machine docker run -it --rm mariadb:10.11.3 mariadb -h 172.31.122.13 -P 3306 -u example-user -p
docker stop some-mariadb
docker rm some-mariadb
docker run --detach --name some-mariadb -p 3306:3306 --env MARIADB_USER=example-user --env MARIADB_PASSWORD=my_cool_secret --env MARIADB_ROOT_PASSWORD=my-secret-pw mariadb:10.11.3
# we also need to open the port on our firewall
sudo ufw allow 3306
Rule added
Rule added (v6)
# You will need to replace the -h <ip_address> with the Private IPv4 of your host machine
docker run -it --rm mariadb:10.11.3 mariadb -h 172.31.122.13 -P 3306 -u example-user -p
Exercise
This exercise will go over a few of the network definitions available within Docker. It will use a busybox container to test various apache web servers.
Chapter 5.5?: Podman vs Docker
The Docker problem
As was mentioned previously, Docker Desktop is now a licensed product. Free for personal use but not something you can use at work without payment. However Docker server maintains its free status but is only available on Linux distributions. This caused a few alternatives to pop up most notably Podman which is sponsored/supported by RedHat.
The Docker Daemon
As you may have noticed, when running docker commands you either need to be a member of the docker group or prepend the commands with sudo so they run as the root user. This is because docker leverages a long running daemon that the command line interface works through. It may not be immediately apparent but because of this design, anybody who wants to run a container through docker has root privileges even if they were only added to the docker group.
The below example illustrates how a new user can see the /etc/shadow file where encrypted passwords (and their salts) are stored!
# Create a new user sudo useradd testuser # Give them the ability to use Docker commands without sudo sudo usermod -aG docker testuser # Switch to that new user sudo su -l testuser # Run a container that suprisingly has root permissions! docker run --rm -it -v /etc:/hostetc ubuntu cat /hostetc/shadow
# Create a new user
sudo useradd testuser
# Give them the ability to use Docker commands without sudo
sudo usermod -aG docker testuser
# Switch to that new user
sudo su -l testuser
# Run a container that suprisingly has root permissions!
docker run --rm -it -v /etc:/hostetc ubuntu cat /hostetc/shadowDaemonless Operation
The Podman approach works slightly differently with the main difference being that podman has no central daemon and no requirement for a user to be added to a specific group. Instead Podman lets any user on the system run containers in their own isolated environment while ensuring the user doesn't gain any additional access they didn't already have. This can cause some confusion and initially unintuitive results for example you may see a different list of container images when running podman image ls vs sudo podman image ls as well as some additional limitations when running without admin/sudo privileges.
Most noteably, non-root (rootless) users cannot mount ports that are less than port 1024 which is where most mainstream services are hosted. Thankfully they can still leverage higher ports like the common 8080 for web development or even PostgreSQL's port 5432. Rootless containers are also limited to accessing the files that the host user can access which is most likely the desired outcome anyway.
Rootless user mapping and permissions.
As we've seen with containers, the user that the container is running as usually does not match the user on the host machine. For example, starting an ubuntu container image will result in a root user prompt regardless of who kicked issued the podman run command. So how is it that podman can isolate different users while still allowing containers to run this way and more importantly, how does this impact file permissions for mounted volumes? Enter the /etc/subuid and /etc/subgid files.
When a user is created on a podman enabled system, an entry is made within the
/etc/subuid file that lists an initial User ID (uid) that would translate to uid=1 followed by a fairly large number indicating how many additional IDs are reserved for that user.
[cloud_user@912cec8e982c ~]$ cat /etc/subuid cloud_user:165536:65536 ...
[cloud_user@912cec8e982c ~]$ cat /etc/subuid
cloud_user:165536:65536
...Podman will translate a User ID back and forth between containers and the host machine by using the information in this file. For example, if a container needs to run as the root user (uid 0), podman will simply save the file as the host user's UID (cloud-user is 1001). For a non-root user ID like the www-data user (UID 33) commonly used in Apache, podman will consult the file /etc/subuid and simply add 33 + 165536 to determine that local files should be 165569 but remember, that files using the root permission don't need to be translated so we start at one less. Our final UID is 165568.
[cloud_user@912cec8e982c ~]$ mkdir test [cloud_user@912cec8e982c ~]$ podman run --rm -it -v $(pwd)/test:/test:Z ubuntu /bin/bash root@b3d55101ea84:/# cd /test root@f5a5959fd656:/test# touch iamroot.txt root@f5a5959fd656:/test# touch webdata root@f5a5959fd656:/test# touch userdata.txt root@f5a5959fd656:/test# chown www-data:www-data webdata root@f5a5959fd656:/test# chown 1000:1000 userdata.txt root@f5a5959fd656:/test# ls -l total 0 -rw-r--r--. 1 root root 0 Nov 16 14:31 iamroot.txt -rw-r--r--. 1 1000 1000 0 Nov 16 14:32 userdata.txt -rw-r--r--. 1 www-data www-data 0 Nov 16 14:31 webdata root@f5a5959fd656:/test# exit exit [cloud_user@912cec8e982c ~]$ ls -l test/ total 0 -rw-r--r--. 1 cloud_user cloud_user 0 Nov 16 14:31 iamroot.txt -rw-r--r--. 1 166535 166535 0 Nov 16 14:32 userdata.txt -rw-r--r--. 1 165568 165568 0 Nov 16 14:31 webdata
[cloud_user@912cec8e982c ~]$ mkdir test
[cloud_user@912cec8e982c ~]$ podman run --rm -it -v $(pwd)/test:/test:Z ubuntu /bin/bash
root@b3d55101ea84:/# cd /test
root@f5a5959fd656:/test# touch iamroot.txt
root@f5a5959fd656:/test# touch webdata
root@f5a5959fd656:/test# touch userdata.txt
root@f5a5959fd656:/test# chown www-data:www-data webdata
root@f5a5959fd656:/test# chown 1000:1000 userdata.txt
root@f5a5959fd656:/test# ls -l
total 0
-rw-r--r--. 1 root root 0 Nov 16 14:31 iamroot.txt
-rw-r--r--. 1 1000 1000 0 Nov 16 14:32 userdata.txt
-rw-r--r--. 1 www-data www-data 0 Nov 16 14:31 webdata
root@f5a5959fd656:/test# exit
exit
[cloud_user@912cec8e982c ~]$ ls -l test/
total 0
-rw-r--r--. 1 cloud_user cloud_user 0 Nov 16 14:31 iamroot.txt
-rw-r--r--. 1 166535 166535 0 Nov 16 14:32 userdata.txt
-rw-r--r--. 1 165568 165568 0 Nov 16 14:31 webdataThe /etc/subgid file represents the exact same concepts for Group IDs (gid). Because of this translation users can leverage containers and still rest assured that if one user makes modifications as a root or service user, it will have no impact on anybody else in the system.
Unshare
Because it would be quite burdensome to do these calculations manually, and still inconvenient to need to create a container every time you want to set up user permissions, podman includes the podman unshare command which will initially look like you ran sudo su -l but actually enters an alternative state where all of the uid and gid translations are active.
[cloud_user@912cec8e982c ~]$ cd test [cloud_user@912cec8e982c test]$ ls -l total 0 -rw-r--r--. 1 cloud_user cloud_user 0 Nov 16 14:31 iamroot.txt -rw-r--r--. 1 166535 166535 0 Nov 16 14:32 userdata.txt -rw-r--r--. 1 165568 165568 0 Nov 16 14:31 webdata [cloud_user@912cec8e982c test]$ podman unshare [root@912cec8e982c test]# ls -l total 0 -rw-r--r--. 1 root root 0 Nov 16 14:31 iamroot.txt -rw-r--r--. 1 1000 1000 0 Nov 16 14:32 userdata.txt -rw-r--r--. 1 33 tape 0 Nov 16 14:31 webdata [root@912cec8e982c test]# exit exit [cloud_user@912cec8e982c test]$
[cloud_user@912cec8e982c ~]$ cd test
[cloud_user@912cec8e982c test]$ ls -l
total 0
-rw-r--r--. 1 cloud_user cloud_user 0 Nov 16 14:31 iamroot.txt
-rw-r--r--. 1 166535 166535 0 Nov 16 14:32 userdata.txt
-rw-r--r--. 1 165568 165568 0 Nov 16 14:31 webdata
[cloud_user@912cec8e982c test]$ podman unshare
[root@912cec8e982c test]# ls -l
total 0
-rw-r--r--. 1 root root 0 Nov 16 14:31 iamroot.txt
-rw-r--r--. 1 1000 1000 0 Nov 16 14:32 userdata.txt
-rw-r--r--. 1 33 tape 0 Nov 16 14:31 webdata
[root@912cec8e982c test]# exit
exit
[cloud_user@912cec8e982c test]$ SELinux integration
Security Enhanced Linux (SELinux) is a Linux security layer created by the NSA that works at the kernel level to limit what files, ports, etc. are accessible by running applications/services. This tool is enabled in Red Hat Enterprise Linux (and its derivatives) by default and is a critical component of STIGS for DoD systems. Other distribution have similar systems like Ubuntu's AppArmor however, they are usually not configured out of the box like SELinux.
Unfortunately the intricacies of SELinux are a bit too much for the purposes of this course but a brief overview of the system is that it is a deny first policy that will explicitly block access to files unless there is an SELinux policy to allow it. SELinux labels files, services, ports, etc. with what is called a context, and the policy defines which process contexts can access which other contexts. The most common configuration is the targeted setup which utilizes 'type contexts' that are easily identified because they end in _t.
[cloud_user@912cec8e982c ~]$ cat /etc/selinux/config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=enforcing # SELINUXTYPE= can take one of these three values: # targeted - Targeted processes are protected, # minimum - Modification of targeted policy. Only selected processes are protected. # mls - Multi Level Security protection. SELINUXTYPE=targeted
[cloud_user@912cec8e982c ~]$ cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these three values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targetedThe ls command can be used to see what SELinux contexts are assigned to files and folders with the -Z flag.
[cloud_user@912cec8e982c Documents]$ touch ~/Documents/test.txt [cloud_user@912cec8e982c Documents]$ ls -Z ~/Documents/test.txt unconfined_u:object_r:user_home_t:s0 /home/cloud_user/Documents/test.txt [cloud_user@912cec8e982c Documents]$ touch /tmp/test2.txt [cloud_user@912cec8e982c Documents]$ ls -Z /tmp/test2.txt unconfined_u:object_r:user_tmp_t:s0 /tmp/test2.txt
[cloud_user@912cec8e982c Documents]$ touch ~/Documents/test.txt
[cloud_user@912cec8e982c Documents]$ ls -Z ~/Documents/test.txt
unconfined_u:object_r:user_home_t:s0 /home/cloud_user/Documents/test.txt
[cloud_user@912cec8e982c Documents]$ touch /tmp/test2.txt
[cloud_user@912cec8e982c Documents]$ ls -Z /tmp/test2.txt
unconfined_u:object_r:user_tmp_t:s0 /tmp/test2.txtContainers and SELinux
Containers don't really follow the traditional SELinux rules because it can quickly become unmanageable to determine which files and services are running on what container for which purpose. While work is being done to make SELinux actually transcend the container boundry, the current approach is to broadly limit containers to accessing only files/directories with the container_file_t label. This label could be assigned with the chcon or semanage with restorecon commands which would normally be used to manage file contexts.
Podman volumes and SELinux
Thankfully podman makes handling SELinux even easier by including a simple flag that can be added to any volume mount. By including :Z at the end of the volume definition, podman will automatically update the SELinux context of the files and folders containerd within the volume making it very simple to mange. We acutally already had to use this in a previous command:
podman run --rm -it -v $(pwd)/test:/test:Z ubuntu /bin/bash ... [cloud_user@912cec8e982c ~]$ ls -Zd ~/test system_u:object_r:container_file_t:s0:c421,c800 /home/cloud_user/test
podman run --rm -it -v $(pwd)/test:/test:Z ubuntu /bin/bash
...
[cloud_user@912cec8e982c ~]$ ls -Zd ~/test
system_u:object_r:container_file_t:s0:c421,c800 /home/cloud_user/testPodman generators
Podman also includes a system of generators that can be used to create a number of repeatable outputs from your podman containers. One of the more useful ones is the ability to generate a systemd unit file so that you can manage your container as if it were any other service running on your system. Additionally, when configured this way, systemd will ensure that the container is running and automatically restart it in the event of an error. In many edge deployment use cases, where high-avalibaility and redundancy are not as important as a low size weight and power footprint, this may be all you need and thankfully everything required to accomplish this is available right out of the box on any RHEL instillation and can be easily installed/configured on other Linux distribution as well. For more information check out man podman-generate-systemd
The podman generate command also has an evolving option to generate kubernetes configurations! Which provides an industry standard approach to defining your full container software stack.
Chapter 6: Understanding Kubernetes
High Level Overview
- Containers run in "Pods" which are the lowest unit within Kubernetes
- A Pod can have multiple "Tightly coupled" containers inside of it
- Kubernetes is managed via definitions of your environment's desired state
- You tell it what you want things to look like
- Kubernetes stores that state in a database (etcd)
- Kubernetes does the best it can to make it so!
Components
On Every Node:
- Kubeadm - A tool that can be used to create a minimum viable Kubernetes cluster that conforms to best practices.
- Kubectl - The Kubernetes command-line tool
- Kublet - An agent that runs on each node in the cluster. It makes sure that containers are running in a Pod.
On Control Nodes:
- kube-apiserver - The API server is a component of the Kubernetes control plane that exposes the Kubernetes API. The API server is the front end for the Kubernetes control plane.
- etcd - Consistent and highly-available key value store used as Kubernetes' backing store for all cluster data.
- kube-scheduler - Control plane component that watches for newly created Pods with no assigned node, and selects a node for them to run on.
- kube-controller-manager - Control plane component that runs controller processes.
- Node controller: Responsible for noticing and responding when nodes go down.
- Job controller: Watches for Job objects that represent one-off tasks, then creates Pods to run those tasks to completion.
- EndpointSlice controller: Populates EndpointSlice objects (to provide a link between Services and Pods).
- ServiceAccount controller: Create default ServiceAccounts for new namespaces.
- cloud-controller-manager - A Kubernetes control plane component that embeds cloud-specific control logic. The cloud controller manager lets you link your cluster into your cloud provider's API, and separates out the components that interact with that cloud platform from components that only interact with your cluster.
Other Addons
- DNS
- Web UI (Dashboard)
- Container Resource Monitoring
- Cluster-level Logging
The CNCF Landscape
https://landscape.cncf.io/
This page shows the sheer volume of tools participating in the Cloud Native Compute Foundation space broken out by their primary focus. Here you can see sections for things like Container Runtimes.
WARN: This page is quite large
Distributions
A Kubernetes distribution is little more than a supported selection of Addons and tools that a company provides.
Some features of Kubernetes drove companies to create pre-configured and supportable Kubernetes Distributions:
- The general popularity of the project
- The OpenSource base Kubernetes is only community supported (Project vs Product)
- The sheer volume of options
- Container Runtime (Docker, Cri-O, rkt, containerd, etc)
- Storage setup (Gluster, Rook, Longhorn, EBS, etc.)
- Netowrking (17+ Options for Networking Policy alone each with their own features and
- Regular updates every 3 months that introduce or sunset features and sometimes introduce bugs.
Some well known Distributions
- Rancher - Offers a range of deployment options
- Rancher Kubenretes Engine (RKE) - A version of Kubernetes that runs entirely within Docker containers
- K3s - An ultra lightweight version of Kubernetes that can run on as little as a raspberry pi but scale easily up to a cluster of large servers
- Red Hat OpenShift
- OKD - The upstream OpenSoruce version of OpenShift
- MicroShift - Upcoming small deployment of Openshift
- Mirantis - Formerly known as Docker Enterprise
- VMWare Tanzu - Focussed on simplifying locally virtualized infrastructure deployments
- Docker kuberentes service (dks) - Included with the enterprise version of Docker. Focussed on reducing the complexity of configuring and running applications under Kubernetes.
- Elastisys compliant kubernetes - Supports the entire application lifecycle and include a security layer
Cloud managed Distributions
- Amazon elastic kubernetes service (eks)
- Google Kubernetes Engine(GKE)
- Azure Kubernetes Service (AKS)
- Rackspace Kubernetes as a service
- DigitalOcean
- Platform9
- Red Hat OpenShift Kubernetes Engine
Installing K3s
K3s is an excellent distribution for our use for a few reasons:
- Quick and easy to get started including everything required to get a cluster running:
- containerd
- Flannel (CNI)
- CoreDNS
- Traefik (Ingress)
- Klipper-lb (Service LB)
- Embedded network policy controller
- Embedded local-path-provisioner
- Host utilities (iptables, socat, etc)
- Can be easily customized to change things like the networking fabric
- Lightweight enough for edge deployments
- Works with SELinux out of the box
- Rancher Federal support options
Some drawbacks:
- In order to be lightweight K3s does not include Kubernetes features that are considered tech previews or deprecated.
- Because k3s is installed via script it can conflict with package manager based security like fapolicyd.
Prepare a node
- Provision a "Medium" Red Hat Enterprise Linux 8 node and wait for it to come up
- Login, change password and do regular updates with
sudo yum -y update
Running the K3s install script
K3 is easily installed via a single highly configurable script. You can run it with or without internet connectivity but for this case we will leverage our connected environment environment it via the below command.
curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE="644" sh -s -
curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE="644" sh -s -The K3S_KUBECONFIG_MODE="644" portion of the above argument makes it so that users on the system outside of the root user can access the default configuration by setting the config file's default permissions to rw-r--r--.
Note about fapolicyd
Because Red Hat Enerprise Linux STIGs require fapolicyd, applications installed outside of yum/dnf will need to either be whitelisted or require custom rules before k3s will function correctly. Some sample rules are below:
# Allow k3s data directory to openly execute untrusted apps within itself
allow perm=execute dir=/var/lib/rancher/k3s/data trust=0 : dir=/var/lib/rancher/k3s/data/ trust=0 ftype=application/x-executable trust=0
# Allow the untrusted k3s application to execute untrusted apps within the k3s data directory
allow perm=execute exe=/usr/local/bin/k3s trust=0 : dir=/var/lib/rancher/k3s/data/ ftype=application/x-executable trust=0
# Allow execute of the untrusted k3s binary by trusted sources
allow perm=execute all trust=1 : path=/usr/local/bin/k3s ftype=application/x-executable trust=0 # Allow k3s data directory to openly execute untrusted apps within itself
allow perm=execute dir=/var/lib/rancher/k3s/data trust=0 : dir=/var/lib/rancher/k3s/data/ trust=0 ftype=application/x-executable trust=0
# Allow the untrusted k3s application to execute untrusted apps within the k3s data directory
allow perm=execute exe=/usr/local/bin/k3s trust=0 : dir=/var/lib/rancher/k3s/data/ ftype=application/x-executable trust=0
# Allow execute of the untrusted k3s binary by trusted sources
allow perm=execute all trust=1 : path=/usr/local/bin/k3s ftype=application/x-executable trust=0Enable Tab Completion
Although our cluster is already operational, a great quality of life feature is the ability to use Tab Completion. Thankfully Kubernetes (and k3s) includes an easy way for us to generate the 300+ line script.
- Install
bash-completionsudo dnf install -y bash-completion
sudo dnf install -y bash-completion - Use
kubectlto create the bash script for you!kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl > /dev/null
kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl > /dev/null - logout and log back Gin
exit
exit
Using Ad-hoc Containers
To get our first container running on the cluster we can use Ad-hoc commands. These commands are typically only used for testing and debugging purposes but provide a quick way for us to make sure our cluster is working properly.
Creating a pod
Ad-Hoc commands are the most familiar way of doing things for those who are accustomed to working with Docker or Podman.
kubectl run --image=nginx web --restart=Never kubectl get pods
kubectl run --image=nginx web --restart=Never
kubectl get podsAll containers are managed as a unit called a Pod which represents one or more "tightly coupled" containers and unless specified otherwise, anything you create will be added to the 'default' namespace
kubectl get namespace # lists the namespaces available on this cluster kubectl get pods # lists pods within the default namespace (unless modified via config) kubectl get pods -n kube-system # lists pods within the 'kube-system' namespace kubectl get pods -A # lists all pods across all namespaces
kubectl get namespace # lists the namespaces available on this cluster
kubectl get pods # lists pods within the default namespace (unless modified via config)
kubectl get pods -n kube-system # lists pods within the 'kube-system' namespace
kubectl get pods -A # lists all pods across all namespacesLooking into pods
The describe command is used to elaborate on almost every type of element within kubernetes
kubectl describe pod web
kubectl describe pod webHere you can see many details about the pod including the IP address and events. The IP address you see here connects directly to this particular instance of this particular container image so if the container were to be created again, the IP address would be different. While you are on a node within the cluster you can access that IP address and even pull the webpage via curl.
## Note the IP address of your web container will be different
curl 10.244.1.3
## or to get the IP and curl in a single line we can use some bash-fooku
curl $(kubectl get pod web --template '{{.status.podIP}}')## Note the IP address of your web container will be different
curl 10.244.1.3
## or to get the IP and curl in a single line we can use some bash-fooku
curl $(kubectl get pod web --template '{{.status.podIP}}')Additionally we can see the output of the container via the logs command. The logs command displays all messages that the application would normally print to the screen via stdout and supports the same flags that tools like less supports so you can display only lines you want or even follow the output and watch live.
Accessing pods interactively
Similar to how Docker lets you connect to a container, you can get a terminal into a pod as well (assuming a shell is present)
kubectl exec -it web -- /bin/bash cat /etc/os_release
kubectl exec -it web -- /bin/bash
cat /etc/os_releaseInjecting a failure
Lets simulate an error in the container by stopping the primary process (pid 1).
kubectl exec web -- /bin/bash -c 'kill 1' kubectl get pod
kubectl exec web -- /bin/bash -c 'kill 1'
kubectl get podNotice how the get pod command shows that 0/1 containers are "READY" and that the status is "Completed." When this is the case, we cannot access the pod like we did before because well.. its gone! This is slightly different from starting and stopping containers in Docker/Podman where you could simply restart it.
The reason the container stays like this is because when we created it, we used the --restart=Never flag. lets create another one slightly differently
kubectl run --image=nginx web2 --restart=Always kubectl get pod kubectl exec web2 -- /bin/bash -c 'kill 1' kubectl get pod
kubectl run --image=nginx web2 --restart=Always
kubectl get pod
kubectl exec web2 -- /bin/bash -c 'kill 1'
kubectl get podThis time the pod returned to a 'Running' state and the "RESTARTS" column now shows a 1 along with when the last restart was in parenthesis.
Cleaning up
To delete our pods we can simply run the delete command
kubectl delete pod web kubectl delete pod web2
kubectl delete pod web
kubectl delete pod web2Moving Toward Declaritive
Running your Kubernetes cluster via ad-hoc commands is tedious, error prone and runs counter to the overall design of containers. Instead, we should aim to run applications in a highly repeatable fashion and take advantage of the idempotence of the container images. Kubernetes is built to manage all aspects of your cluster declaratively meaning that you simply 'declare' a desired state of your pods and other aspects in your cluster, and the kubelet will ensure that it does its best to achieve it.
Common parts of a deployment
One way to declare something is via a deployment which contains multiple items:
- The Deployment object itself
- A ReplicaSet to define how many instances of a pod we want to have running at a given time
- A template: to outline how we want the replicated pods to look like
- The containers themselves defined within the pod
Kubernetes can use a few markup languages to define a deployment but most the most common is yaml. an example deployment yaml is below:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80Creating a Deployment
- Copy the above text into a file called
mydeployment.yamlusing your text editor of choice. - Create the contents of the file with the
kubectl createcommandkubectl create -f mydeployment.yaml kubectl get pods
kubectl create -f mydeployment.yaml kubectl get pods
The deployment will create pods with a semi-random format of <deployment_name>-<random_replicaset_identifier>-<random_pod_identifier> such as nginx-deployment-cbdccf466-cnsf2.
Notice that unlike in the past, we immediately have three pods. This is because we defined replicas: 3 in our deployment definition and Kubernetes will make sure that happens even if one of the pods goes away.
# Delete one of your pods kubectl delete pod <full pod name> # Or use bash to just pick the first one from the list kubectl delete $(kubectl get pods -o=NAME | head -1) kubectl get pod
# Delete one of your pods
kubectl delete pod <full pod name>
# Or use bash to just pick the first one from the list
kubectl delete $(kubectl get pods -o=NAME | head -1)
kubectl get podNotice that we still have three running pods but one of them has a significantly more recent AGE. This is because Kubernetes noticed that we don't have as many as we requested and created a new one for us! This is a clear advantage to making sure applications are highly available and able to scale to fit demand.
Take a look around
There are many parts to Kubernetes beyond just the pod and as we've seen in this section, you can use them to create some interesting behaviors. We can see some of the new components we've described in this chapter similarly to how we did with the pod get and describe statements
kubectl deployments kubectl describe deployments nginx-deployment kubectl get deployments -A kubectl get replicasets kubectl describe $(kubectl get replicasets -o=NAME) kubectl get replicasets -A
kubectl deployments
kubectl describe deployments nginx-deployment
kubectl get deployments -A
kubectl get replicasets
kubectl describe $(kubectl get replicasets -o=NAME)
kubectl get replicasets -AOther API Resources
There are actually many resources within Kubernetes by default and even more that can be added via Custom Resource Definitions (crds) but you can list all available resources, their KIND which is used in the yaml definitions and even if this it can be used in a namespace or not via the kubectl api-resources command.
kubectl api-resources
kubectl api-resourcesWhat is even nicer is that you can get an abbreviated guide of the resource's fields using the explain command and providing a resource.
kubectl explain configmap
kubectl explain configmapModify a deployment
In a typical production Kubernetes environment, all of the yaml files, known as manifests, are version controlled and deployment of them would go through a pipeline however, in our test environment we can simply modify the file and send it to kubernetes. For now, lets modify the replicas of the nginx deployment we already have.
- Update the mydeployment.yaml file via a text editor and change the
replicas:parameter to10. - Use the
applycommand to tell kuberentes about our change.kubectl apply -f mydeployment.yaml
kubectl apply -f mydeployment.yaml - observe the changes
kubectl get pods
kubectl get pods
Scaling a deployment
Because scaling a is fairly common, there is a direct command we can use called scale as well as automated techniques that are beyond the scope of this course.
kubectl scale deployment nginx-deployment --replicas 2 kubectl get pods
kubectl scale deployment nginx-deployment --replicas 2
kubectl get podsCleaning up with a manifest
Because a manifest file can contain a lot of different things, it might be difficult to remove it manually. Thankfully there is a very easy way to accomplish this by once again providing the file.
kubectl delete -f mydeployment.yaml
kubectl delete -f mydeployment.yamlThis does however, assume you haven't modified the components defined within the file sense you initially used it with create
NOTE: it is usually a best practice to create application namespaces and keep relevant resources together. This makes cleanup very easy because deleting a namespace will trigger the cleanup of everything defined within it. This also makes it significantly less likely that resources inadvertently conflict or get modified.
Chapter 7: Advanced Kubernetes
Kubernetes services
So far we've been able to work with pod IP addresses and access them from within our cluster however, thats not very useful for publishing applications like a web page. To make things even more difficult, when a pod is created it will be assigned a semi-random IP address and this IP address is very likely to change when applications crash, get moved to new worker nodes, and scale. For this reason, kubernetes offers a type of resource called a Service.
A Kubernetes Service is a resource specifically designed to expose a network to one or more Pods in your cluster. It will automatically discover and track pods as they move around or their IP addresses change making it so you no longer need to manually look up what the internal IP of the container is and you can simply refer to the IP address of the service above it.
Defining a Service
Similarly to how we created a Deployment, we will define a service in a manifest file too. An example service is below:
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
selector:
app.kubernetes.io/name: MyApp
ports:
- protocol: TCP
port: 80
targetPort: 9376apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
selector:
app.kubernetes.io/name: MyApp
ports:
- protocol: TCP
port: 80
targetPort: 9376In this instance we have a service my-service that connects to pods that have the name: MyApp such that incoming traffic on port 80 is redirected to the container's port 9376 . This style of port mapping is similar to the docker/podman -p 80:9376 flag.
Labels
With deployments and now services we've seen a method of naming things within the metadata: section. This is how Kubernetes identifies objects and is loosely defined with a key: value syntax so that you can create any label you need.
Using the previous nginx example where we have the below block, we can see that the Deployment has a labels section including a key/value of app: nginx
metadata:
name: nginx-deployment
labels:
app: nginxmetadata:
name: nginx-deployment
labels:
app: nginxUsing this information, we can create a service that routes to the correct applications and ports. For illustration purposes we will route the default http port (port 80) of the container to the service's port 8080.
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
selector:
app: nginx
ports:
- protocol: TCP
port: 8080
targetPort: 80apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
selector:
app: nginx
ports:
- protocol: TCP
port: 8080
targetPort: 80Notice the label of the deployment is listed within the service's selector: section. We can create this service either by creating new manifest file for the service, or we can put in our previous mydeployment.yaml file by using the (previously optional) yaml --- and ... syntax to identify that we are starting a new document.
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
...
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
selector:
app: nginx
ports:
- protocol: TCP
port: 8080
targetPort: 80
...---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
...
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
selector:
app: nginx
ports:
- protocol: TCP
port: 8080
targetPort: 80
...Testing our service
- Modify or create the
mydeployment.yamlwith the previous example - Apply the changes
kubectl apply -f mydeployment.yaml
kubectl apply -f mydeployment.yaml - Inspect the serviceNotice the list of IP addresses and port numbers under
kubectl get service kubectl describe service nginx-service
kubectl get service kubectl describe service nginx-serviceEndpoints:. These correspond to the available pods within your cluster. Also notice that the service itself has an IP address. - Verify service endpoints correspond to your pod IPs.
kubectl get pods -o wide
kubectl get pods -o wide - Try to access the service IP on port 8080
curl <IP_Of_Service>:8080 # Or to dynamically find and use it curl $(kubectl get service/nginx-service -o jsonpath='{.spec.clusterIP}'):8080curl <IP_Of_Service>:8080 # Or to dynamically find and use it curl $(kubectl get service/nginx-service -o jsonpath='{.spec.clusterIP}'):8080
In this case we are seeing the output of a single pod from within our deployment. The Service will spread requests around all of the currently Running pods and avoid any that aren't ready to receive traffic! This lets safely accomplish deployments while keeping application availability.
Demonstration
Using tmux, we can observe the service uptime, pod deployments and what happens if we update the container image we are using!
Common Service types
There are three primary service types in Kubernetes that create different behaviour. The default service type is ClusterIP which is what our previous example used. A ClusterIP service is useful for exposing an application to other applications within your cluster but cannot be seen externally. The full list of service types include:
- ClusterIP - Only reachable from within the cluster
- NodePort - Exposes the Service on each Node's IP address at a static port.
- LoadBalancer - Exposes the service externally using a load balancer. These must be configured and are often provided by cloud service providers like AWS.
- ExternalName - Maps the service to a DNS name as listed with the
externalName:parameter. This configures the cluster's DNS service to return aCNAMErecord.
Lets modify the service we previously created and see if we can't access the webpage from a browser!
- Edit the service manifest as below:
--- apiVersion: v1 kind: Service metadata: name: nginx-service spec: selector: app: nginx type: NodePort ports: - protocol: TCP port: 8080 targetPort: 80--- apiVersion: v1 kind: Service metadata: name: nginx-service spec: selector: app: nginx type: NodePort ports: - protocol: TCP port: 8080 targetPort: 80 - Find the correct host port that the service was mapped to.
Because multiple services can be created this way, Kubernetes will assign a random Node port to the service from a given range (default: 30000-32767). You can identify this port by simply listing the services.kubectl get service nginx-service
kubectl get service nginx-service
Using an ingress
An ingress is a Kubernetes resource that acts as a reverse proxy to internal ClusterIP type services. This can greatly simplify a host and reduce the number of IP addresses a company needs to request from their service provider.
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: my-ingress
spec:
routes:
- match: PathPrefix(`/cerf`)
kind: Rule
services:
- name: nginx-service
port: 8080
middlewares:
- name: flaskapp-stripprefix
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: flaskapp-stripprefix
spec:
stripPrefix:
prefixes:
- /cerf---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: my-ingress
spec:
routes:
- match: PathPrefix(`/cerf`)
kind: Rule
services:
- name: nginx-service
port: 8080
middlewares:
- name: flaskapp-stripprefix
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: flaskapp-stripprefix
spec:
stripPrefix:
prefixes:
- /cerfHigh availability Demonstration
In this demonstration we will use a container that is designed to fail after 3 website hits and observe how Kubernetes load balancing and auto recovering features establish high availability.
- Deploy the following manifest:
cat << EOF >> ~/hellogo_deployment.yaml --- apiVersion: v1 kind: Namespace metadata: name: hellogo-ns --- apiVersion: apps/v1 kind: Deployment metadata: name: hellogo-deployment namespace: hellogo-ns labels: app: hellogo spec: replicas: 1 selector: matchLabels: app: hellogo template: metadata: labels: app: hellogo spec: containers: - name: hellogo image: simplicitie/katacoda-hellogo:latest env: - name: NAME value: "Friends" - name: FAILAFTER value: "3" ports: - containerPort: 3000 --- apiVersion: v1 kind: Service metadata: name: hellogo-service namespace: hellogo-ns spec: selector: app: hellogo ports: - protocol: TCP port: 80 targetPort: 3000 --- apiVersion: traefik.containo.us/v1alpha1 kind: IngressRoute metadata: name: hellogo-ingress namespace: hellogo-ns spec: routes: - match: PathPrefix(`/ha`) kind: Rule services: - name: hellogo-service port: 80 middlewares: - name: ha-stripprefix --- apiVersion: traefik.containo.us/v1alpha1 kind: Middleware metadata: name: ha-stripprefix namespace: hellogo-ns spec: stripPrefix: prefixes: - /ha EOFcat << EOF >> ~/hellogo_deployment.yaml --- apiVersion: v1 kind: Namespace metadata: name: hellogo-ns --- apiVersion: apps/v1 kind: Deployment metadata: name: hellogo-deployment namespace: hellogo-ns labels: app: hellogo spec: replicas: 1 selector: matchLabels: app: hellogo template: metadata: labels: app: hellogo spec: containers: - name: hellogo image: simplicitie/katacoda-hellogo:latest env: - name: NAME value: "Friends" - name: FAILAFTER value: "3" ports: - containerPort: 3000 --- apiVersion: v1 kind: Service metadata: name: hellogo-service namespace: hellogo-ns spec: selector: app: hellogo ports: - protocol: TCP port: 80 targetPort: 3000 --- apiVersion: traefik.containo.us/v1alpha1 kind: IngressRoute metadata: name: hellogo-ingress namespace: hellogo-ns spec: routes: - match: PathPrefix(`/ha`) kind: Rule services: - name: hellogo-service port: 80 middlewares: - name: ha-stripprefix --- apiVersion: traefik.containo.us/v1alpha1 kind: Middleware metadata: name: ha-stripprefix namespace: hellogo-ns spec: stripPrefix: prefixes: - /ha EOF - Using
tmux, open three windows- One to
watch kubectl pods -n hellogo-ns - One to view the
logsof a pod and see when the page is being hit - Another to run curl commands against the website
/haprefix. Alternatively you could navigate via a web browser.
- One to
Persistent Storage
While many applications don't need to store data, there are some that are reliant on it! For example, a database, message queue or file server and while they will work within memory by default, we need the data to persist pod deletion/creation.
Storage within Kubernetes is initially complex but that is because it is designed to be not only extremely flexible but to draw a definitive line between what a system administrator would configure and what a developer needs. This abstraction makes it fairly easy to migrate between entirely different Kubernetes clusters without the need to know how exactly things are configured.
The Parts of Kubernetes storage
- Provisioner - This is the underlying tool used to establish the storage. Your cluster Admin or cloud hosting environment will usually provide this layer.
- storageClass - A Provisioner can establish one or more storageClasses which contains some settings like
reclaimPolicy:which defines what to do with unused volumes,volumeBindMode:to define when the volume should be created, and more. - persistentVolume - Defines the space being reserved including the size of the storage.
- persistentVolumeClaim - Connects the PV to a container
Exercise
- Identify your StroageClass
kubectl get storageclass kubectl describe storageclasses local-path
kubectl get storageclass kubectl describe storageclasses local-path - Create a PersistentVolumeClaima. Apply the above yaml file
kind: PersistentVolumeClaim apiVersion: v1 metadata: name: my-pv-claim spec: storageClassName: local-path accessModes: - ReadWriteOnce resources: requests: storage: 1Gikind: PersistentVolumeClaim apiVersion: v1 metadata: name: my-pv-claim spec: storageClassName: local-path accessModes: - ReadWriteOnce resources: requests: storage: 1Gi
b. Observe the results when runningkubectl get pvcandkubectl get pvandkubectl describe pvc my-pv-claim. Notice that the pvc is "waiting for first consumer to be created" before it creates thepvand binds to it. - Create a pod that uses the PVCIn this pod, we are using a simple sleep statement to ensure the pod does not report that it is "complete" and shut down before we are ready. This is a fairly common technique and is often used with containers like busybox.
apiVersion: v1 kind: Pod metadata: name: pvc-user spec: containers: - name: pvc-user image: ubuntu command: [ "/bin/bash", "-c", "--" ] args: [ "while true; do sleep 30; done;" ] volumeMounts: - mountPath: "/stuff" name: my-stuff volumes: - name: my-stuff persistentVolumeClaim: claimName: my-pv-claimapiVersion: v1 kind: Pod metadata: name: pvc-user spec: containers: - name: pvc-user image: ubuntu command: [ "/bin/bash", "-c", "--" ] args: [ "while true; do sleep 30; done;" ] volumeMounts: - mountPath: "/stuff" name: my-stuff volumes: - name: my-stuff persistentVolumeClaim: claimName: my-pv-claim
a. Apply the above yaml file
b. Now observe the results of the previous commands and notice that a PV has been created.Notice that the PV has been created and bindings for the pvc are establishedkubectl get pvc kubectl get pv
kubectl get pvc kubectl get pv - Enter the pod and create some example files
kubectl exec -it pvc-user -- /bin/bash cd stuff printf "1. A Pony\n2. Peace on Earth" > x-mas-list.txt echo "My Awsome Homepage" > index.html ls
kubectl exec -it pvc-user -- /bin/bash cd stuff printf "1. A Pony\n2. Peace on Earth" > x-mas-list.txt echo "My Awsome Homepage" > index.html ls - Now we will delete the pod and connect the same pvc to another Pod. For this exercise we will use nginx and display our files to curl:
kubectl delete pod pvc-user cat << EOF >> ~/pvc-web.yaml apiVersion: v1 kind: Pod metadata: name: pvc-web spec: containers: - name: pvc-web image: nginx volumeMounts: - mountPath: "/usr/share/nginx/html" name: mypvc volumes: - name: mypvc persistentVolumeClaim: claimName: my-pv-claim EOF kubectl apply -f pvc-web.yamlkubectl delete pod pvc-user cat << EOF >> ~/pvc-web.yaml apiVersion: v1 kind: Pod metadata: name: pvc-web spec: containers: - name: pvc-web image: nginx volumeMounts: - mountPath: "/usr/share/nginx/html" name: mypvc volumes: - name: mypvc persistentVolumeClaim: claimName: my-pv-claim EOF kubectl apply -f pvc-web.yaml - Using curl we can now see what our web server is hosting!
curl $(kubectl get pod pvc-web --template '{{.status.podIP}}') curl $(kubectl get pod pvc-web --template '{{.status.podIP}}')/x-mas-list.txtcurl $(kubectl get pod pvc-web --template '{{.status.podIP}}') curl $(kubectl get pod pvc-web --template '{{.status.podIP}}')/x-mas-list.txt
StatefulSets
https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/
Similar to a Deployment, a StatfulSet defines a template for pods and a replicaset however, StatfulSets have one important difference. Lets assume your application is highly scaleable that stores data within a volume. Every time a container comes online it joins a "cluster" of application containers (or pods) by joining and syncronizing data. If an error occurs and one of the application containers goes offline, a tranidtional deployment would randomly generate another one and join the cluster however, over time your application cluster would have a potentally endless number of nodes within it that are offline for every pod that was ever started.
StatefulSets are different in that they ensure that pods are created in a sequential and consistent order with a predictiable identifier and volume match (sticky identity). This means that as you scale from 1 replica to 3, the statefulset will first ensure that pod #2 is online and active before it starts to provision #3. Additionally a volume will be created and assigned to each pod sequentially. This way if a container goes down, a new container will be built and assigned that same volume and name providing a much cleaner approach.
StatefulSets are the recommended way of deploying pods that leverage volumes if you ever intend to scale beyond a single replica.
Cassandra example
https://kubernetes.io/docs/tutorials/stateful-application/cassandra/
NOTE: Be sure to set up the service first with the spec of clusterIP: None. This ensures that the networkign stayts consistent between pods and alows them to easily find one another.
apiVersion: v1
kind: Service
metadata:
labels:
app: cassandra
name: cassandra
spec:
clusterIP: None
ports:
- port: 9042
selector:
app: cassandraapiVersion: v1
kind: Service
metadata:
labels:
app: cassandra
name: cassandra
spec:
clusterIP: None
ports:
- port: 9042
selector:
app: cassandraNOTE: when creating the StatefulSet, you will need to modify the yaml file slightly so that the provisioner: parameter of the StorageClass so that it reflects the storage class on our system. IE provisioner: local-path
[cloud_user@912cec8e983c cassandra]$ kubectl get storageclass NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 28m [cloud_user@912cec8e983c cassandra]$ wget https://k8s.io/examples/application/cassandra/cassandra-statefulset.yaml [cloud_user@912cec8e983c cassandra]$ vim cassandra-statefulset.yaml
[cloud_user@912cec8e983c cassandra]$ kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 28m
[cloud_user@912cec8e983c cassandra]$ wget https://k8s.io/examples/application/cassandra/cassandra-statefulset.yaml
[cloud_user@912cec8e983c cassandra]$ vim cassandra-statefulset.yaml NOTE: you will need to update the CASSANRDA SEED to point to the correct namespace where your pod resides by replacing the default portion of below environment variable with your namespace name:
- name: CASSANDRA_SEEDS
value: "cassandra-0.cassandra.default.svc.cluster.local" - name: CASSANDRA_SEEDS
value: "cassandra-0.cassandra.default.svc.cluster.local"ConfigMaps
https://kubernetes.io/docs/concepts/configuration/configmap/
Secrets
https://kubernetes.io/docs/concepts/configuration/secret/
Helm
As you have seen, Kubernetes setups can have a number of pieces/parts and we've only covered a handful of the main ones in this class. Manifest (yaml) files are great for making sure that our deployments are repeatable, and version controlled however, they are challenging to update and/or tweak when we need to make minor modifications between environments. This is not ideal for some use cases like a software company trying to release their cloud native product to the public.
Helm fits this gap by acting as a sort of package manager for Kubernetes deployment where the entire system is defined in what is called a "chart". Charts are actually relatively simple and consist mostly of a Chart.yaml file with metadata about the cart itself, a values.yaml file that contains a list of variables that can be configured for the chart and a folder of templates that describe various Kubernetes resources with slightly modified .yaml files.
A Helm template contains of the same elements for any other resource with the simple addition of variables contained within the {{ }} characters. These variables usually come from the values.yaml file for example:
ports:
- name: cql
port: {{ .Values.service.ports.cql }} ports:
- name: cql
port: {{ .Values.service.ports.cql }}Templates can also leverage conditional statements like an {{- if <some condition>}}, {{-else if <some_condition>}}, {{-else }}, {{- end }} statment.
A Helm chart can even include other charts making for a fairly extensible and widely adopted system.
Using Helm
While k3s already includes helm, we can go through the standard instillation process just the same. However first, we must ensure that we have properly established our user's ~/.kube/config file. Up until now that has been stored in /etc/rancher/k3s/k3s.yaml we can set this up with the below commands:
cp /etc/rancher/k3s/k3s.yaml ~/.kube/config chmod 700 ~/.kube/config
cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
chmod 700 ~/.kube/configAt this point you can run any of the normal helm commands including the quick-start-guide which demonstrates how to install a mysql release.
https://helm.sh/docs/intro/quickstart/
Chapter 8: GoCount Project
In this chapter we will bring put together everything we've learned in support of containerizing an application and running it within our cloud environment! In this scenario a developer created a simple application to bring Wallmart/Costsco door greeters into the modern era!
GoCount is a very small application written in the Go programming language that exposes simple methods like incr (short for increment) to add 1 to the counter and reset to set the count back to 0. However, to better futureproof the application in case a store has multiple entrance points, the developer stored the current count in a Redis database. Our task is to containerize the application and host it within a Kubernetes environment.
Part 1: Dockerize the app
The application code can be found at https://github.com/Simplicitie/gocount and includes a README file that explains how the application works, how to compile it and how to configure the Redis connection information.
Create a Redis Database
Thankfully Redis is a common database and is available on DockerHub. Please Create a local REDIS instance using the official container image available at https://hub.docker.com/_/redis.
- Create a Redis Container in either your Docker or Podman environment such that the redis database is available on
localhost:6379
NOTES:
- For a smaller download and container look for an
alpineimage - To test the container, use a second Redis container.
-
You will need to know the IP address of the first container.
-
A Redis container can use the
redis-clicommand to get an interactive command line. This command takes the-hflag for hostname or IP address to connect to. -
Redis is a an in-memory key-value store and some basic commands you can use to test that it works are
SETandGET10.88.0.12:6379> SET foo 4 OK 10.88.0.12:6379> GET foo "4" 10.88.0.12:6379> SET bar "test" OK 10.88.0.12:6379> GET bar "test" 10.88.0.12:6379> GET HiMom! (nil) 10.88.0.12:6379> exit
10.88.0.12:6379> SET foo 4 OK 10.88.0.12:6379> GET foo "4" 10.88.0.12:6379> SET bar "test" OK 10.88.0.12:6379> GET bar "test" 10.88.0.12:6379> GET HiMom! (nil) 10.88.0.12:6379> exit
-
Create a Dockerfile that builds the GoCount application
Now that we have a database established, we can build, run, and test our application. please create a Dockerfile or Containerfile that compiles and runs the app.
- The Containerfile should have instructions to include the
main.gofile and compile/build it per the instructions in theREADME.mdfile on https://github.com/Simplicitie/gocount. - Test the container by running it and accessing http://localhost:8080 via
curl http://localhost:8080/incr.
NOTES:
-
Instead of installing and congiguring Go, you can use the golang image available at https://hub.docker.com/_/golang
-
You will either need to include a
go.modfile or generate one prior to building with two simple commands.go mod init gocount go mod tidy
go mod init gocount go mod tidy -
Because the build process creates a single executable which is all we really are about, you may consider using a multi-stage build and moving the compiled executable into a
scratchcontainer. -
The GoCount application has some convenient default values for connecting to a local Redis instance however, with Redis being in another container, you may need to provide the Container Name or IP address to GoCount via the
REDIS_HOSTenvironment variable when running.
Creating a Compose file
While the application works now within a container, it is far too error prone to set up. Please create a docker-compose.yaml that will set up and configure the full application stack
- The compose file should include instructions for deploying both Redis and GoCount
- The compose file should expose port 8080 of GoCount so that the application can be accessed.
- The compose file should create an isolated network between Redis and GoCount so that the Database is only visible to containers that need it. This will also provide a DNS feature so that we can use names instead of IP Addresses.
NOTES:
- If you were unable to build the container locally, it can be pulled from Docker Hub at https://hub.docker.com/r/simplicitie/gocount
- You will likely need to leverage Google and the [Compose v3 Documentation]](https://docs.docker.com/compose/compose-file/compose-file-v3/)
- If you are working within a podman environment, you will need to install [podman-compose](https://github.com/containers/podman-compose
- For a bonus, see if you can get the count to persist a redeployment of the Redis Container. This would require a volume that, per the Redis docs, should be mounted to
/data
Part 2: Run within Kubernetes!
Create a Redis Pod
While docker is great, if GoCount is ever going to be adopted enterprise wide, it needs a much more scalable deployment! With Kubernetes, we rest assured that the system can be deployed to a wide range of managed services across cloud providers. as with before, we need to start by tackling the Redis dependency.
- Create a namespace called
gocountto hold all our resources. - Create a Deployment that manages pods so that Kubernetes can monitor the availability.
NOTES:
- Feel free to use
kubectl create ... --dry-run=client -o yamlto create a starting point.
Create a Service for the Redis deployment
-
The service should use port 6379 via TCP
-
Test connectivity to the Redis pod(s) via the service by running an ad-hoc Kubernetes container
# This assumes the gocount namespace is being use and that the service is named "redis" kubectl run -it -n gocount --image redis:7.2-alpine test -- /bin/ash # /data # redis-cli -h redis`
# This assumes the gocount namespace is being use and that the service is named "redis" kubectl run -it -n gocount --image redis:7.2-alpine test -- /bin/ash # /data # redis-cli -h redis`
Create the GoCount Deployment
- Use the container image "simplicitie/gocount" hosted on DockerHub
- You will need to provide an environment variable to the container named REDIS_HOST and set it to the name of the Redis service we created previously
- To verify the container is running, simply check the logs of the pod. If it has the line [GIN-debug] Listening and serving HTTP on :8080 you are good to go and connected to Redis!
Enable data persistence in Redis
- Create a PVC Object
- Modify your Redis deployment to mount the PVC as a volume in the
/datapath - Test that your count application still retains the last known number after Redis is deleted and recreated
NOTES:
- PVC documentation can be found at https://kubernetes.io/docs/concepts/storage/persistent-volumes/