|
|
|
@@ -0,0 +1,165 @@ |
|
|
|
--- |
|
|
|
title: Isolated IP Management |
|
|
|
description: > |
|
|
|
For routers, a management interface should always work, even if the |
|
|
|
firewall is broken, or the IP address conflicts with other interfaces. |
|
|
|
Isolating the interface to a vnet jail and using unix domain sockets fixes |
|
|
|
this problem. |
|
|
|
posted: !!timestamp '2023-08-17' |
|
|
|
created: !!timestamp '2023-08-17' |
|
|
|
time: 12:00 PM |
|
|
|
tags: |
|
|
|
- FreeBSD |
|
|
|
- jails |
|
|
|
--- |
|
|
|
|
|
|
|
The latest iteration of my home firewall has a spare interface. Many |
|
|
|
switches these days have a dedicated management interface, and I wanted |
|
|
|
something similar for my firewall. I want an interface that I can plug in, |
|
|
|
get an IP via DHCP, and I can ssh into and it'll "just work" even if |
|
|
|
there's a misconfiguration, or an IP conflict. |
|
|
|
|
|
|
|
On FreeBSD, |
|
|
|
[vnet jails](https://wiki.freebsd.org/Jails#VNET-based_networking_for_Jails) |
|
|
|
are a way to isolate an interface such that it won't interfere (or receive |
|
|
|
interference) from other interfaces. Some may have just used epair and |
|
|
|
assigned IPs, but this could cause conflict, while using a unix domain |
|
|
|
socket keeps things isolated, and FreeBSD ships w/ everything needed to |
|
|
|
make this work. A package like `socat` isn't needed. |
|
|
|
|
|
|
|
Configuring the host was straight forward, make a directory for the socket: |
|
|
|
``` |
|
|
|
mkdir /var/mgmt |
|
|
|
``` |
|
|
|
|
|
|
|
And then add a line to `/etc/inetd.conf` to listen to incoming connections |
|
|
|
on the socket and launch `sshd`: |
|
|
|
``` |
|
|
|
/var/mgmt/mgmt.ssh.sock stream unix nowait root /usr/sbin/sshd sshd -i |
|
|
|
``` |
|
|
|
|
|
|
|
The `-i` option to `sshd` tells it to run in "`inetd`" mode, which means that |
|
|
|
the stdin and stdout of the process is the socket to use for communication. |
|
|
|
|
|
|
|
The next harder part was getting a jail configured that would accept |
|
|
|
incoming connections and forward them to the unix domain socket. |
|
|
|
|
|
|
|
First the jail configuration, which goes in `/etc/jail.conf` or similar |
|
|
|
location: |
|
|
|
``` |
|
|
|
mgmt { |
|
|
|
host.hostname = mgmt; # Hostname |
|
|
|
|
|
|
|
vnet ="new"; |
|
|
|
vnet.interface="mgmt0"; |
|
|
|
|
|
|
|
path = "/usr/jails/mgmt/root"; # Path to the jail |
|
|
|
mount.fstab="/usr/jails/mgmt/fstab"; # mount spec |
|
|
|
|
|
|
|
mount.devfs; # Mount devfs inside the jail |
|
|
|
devfs_ruleset = "101"; |
|
|
|
|
|
|
|
exec.start = "/bin/sh /etc/rc"; # Start command |
|
|
|
exec.stop = "/bin/sh /etc/rc.shutdown"; # Stop command |
|
|
|
} |
|
|
|
``` |
|
|
|
|
|
|
|
There isn't anything special in this. It's pretty standard jail |
|
|
|
configuration, the differences are the vnet configuration, and the |
|
|
|
`devfs_ruleset`. |
|
|
|
|
|
|
|
The `devfs_ruleset` is necessary in order to expose the bpf interface |
|
|
|
used by dhclient. This required the following lines in `/etc/devfs.rules`: |
|
|
|
``` |
|
|
|
[mydevfsrules_jail=100] |
|
|
|
add include $devfsrules_hide_all |
|
|
|
add include $devfsrules_unhide_basic |
|
|
|
add include $devfsrules_unhide_login |
|
|
|
|
|
|
|
[devfsrules_jail_dhcp=101] |
|
|
|
add include $mydevfsrules_jail |
|
|
|
add path 'bpf*' unhide |
|
|
|
``` |
|
|
|
|
|
|
|
Note that after adding the above lines, you need to run: |
|
|
|
``` |
|
|
|
service devfs start |
|
|
|
``` |
|
|
|
to load the rules (per |
|
|
|
[devfs(8)](https://man.freebsd.org/cgi/man.cgi?query=devfs&apropos=0&sektion=8&manpath=FreeBSD+13.2-RELEASE+and+Ports&arch=default&format=html) |
|
|
|
man page). |
|
|
|
|
|
|
|
Note: I learned my mistake not to number my blocks immediately after |
|
|
|
the standard defaults (from `/etc/defaults/devfs.rules`). I had done that |
|
|
|
befure but there's now a conflict, so I skip ahead a bit to get a unique range. |
|
|
|
|
|
|
|
Now I needed to make a number of directories for the jail: |
|
|
|
``` |
|
|
|
mkdir -p /usr/jails/mgmt/root |
|
|
|
mkdir -p /usr/jails/mgmt/etc |
|
|
|
mkdir -p /usr/jails/mgmt/var/mgmt |
|
|
|
mkdir -p /usr/jails/mgmt/tmp |
|
|
|
``` |
|
|
|
|
|
|
|
I needed to setup the `fstab` for the jail: |
|
|
|
``` |
|
|
|
# Device Mountpoint FStype Options Dump Pass# |
|
|
|
/ /usr/jail/mgmt/root nullfs ro 0 0 |
|
|
|
/usr/jail/mgmt/etc /usr/jail/mgmt/root/etc nullfs rw 0 0 |
|
|
|
/usr/jail/mgmt/var /usr/jail/mgmt/root/var nullfs rw 0 0 |
|
|
|
/var/mgmt /usr/jail/mgmt/root/var/mgmt nullfs ro 0 0 |
|
|
|
/usr/jail/mgmt/tmp /usr/jail/mgmt/root/tmp nullfs rw 0 0 |
|
|
|
``` |
|
|
|
|
|
|
|
This is a little bit more tricky, It first `nullfs` mounts the root system. |
|
|
|
I'm using ZFS boot environments, so this is a pretty clean FreeBSD install |
|
|
|
without much host specific data. It then mounts some jail specific directories |
|
|
|
for `etc`, `tmp` and `var` and finally mounts the shared directory w/ the |
|
|
|
unix domain socket to the host system. Also note that a couple of the mounts |
|
|
|
are read-only to prevent the jail from modifying the system. |
|
|
|
|
|
|
|
The `etc` directory was populated from the system `etc` via: |
|
|
|
``` |
|
|
|
tar -cf - -C /etc . | tar -xf - -C /usr/jails/mgmt/etc |
|
|
|
``` |
|
|
|
|
|
|
|
Then the jail was configured, first `/usr/jails/mgmt/etc/rc.conf`: |
|
|
|
``` |
|
|
|
hostname="mgmt.funkthat.com" |
|
|
|
|
|
|
|
sendmail_enable="NONE" |
|
|
|
sendmail_submit_enable="NO" |
|
|
|
sendmail_outbound_enable="NO" |
|
|
|
sendmail_msp_queue_enable="NO" |
|
|
|
|
|
|
|
# Management port |
|
|
|
ifconfig_mgmt0="DHCP" |
|
|
|
|
|
|
|
# necessary as devd can't be run in a jail |
|
|
|
synchronous_dhclient="YES" |
|
|
|
|
|
|
|
inetd_enable="YES" # Run the network daemon dispatcher (YES/NO). |
|
|
|
``` |
|
|
|
|
|
|
|
The key part of this configuration that took me a while to figure out was |
|
|
|
the `synchronous_dhclient` line. It used to be that `netif` would start |
|
|
|
dhclient, but in order to better handle USB ethernet devices and other |
|
|
|
removable interfaces, it was moved to `devd`. The only problem is that |
|
|
|
`devd` hasn't been jail'ified, and you can't run it to get things like |
|
|
|
link notifications that would normally launch dhclient. Setting this to |
|
|
|
yes, makes sure it gets launched when the jail starts. |
|
|
|
|
|
|
|
And then the following line was added to `/usr/jails/mgmt/etc/inetd.conf`: |
|
|
|
``` |
|
|
|
ssh stream tcp nowait root /usr/bin/nc nc -N -U /var/mgmt/mgmt.ssh.sock |
|
|
|
``` |
|
|
|
|
|
|
|
This is the part that will forward incoming connections to the ssh port |
|
|
|
on to the unix domain socket. |
|
|
|
|
|
|
|
Now that everything is configured, a simple `jail -c mgmt` will get |
|
|
|
the jail running and accepting connections. |
|
|
|
|
|
|
|
This was testing and deployed on a FreeBSD 14-CURRENT build as of August |
|
|
|
8th, 2023, or more specifically, from `main-n264621-09c20a29328`, but it |
|
|
|
should work on all currently supported FreeBSD releases. |
John-Mark Gurney