blog/content/posts/isolated-ip-management.html

---
title: Isolated IP Management
description: >
  For routers, a management interface should always work, even if the
  firewall is broken, or the IP address conflicts with other interfaces.
  Isolating the interface to a vnet jail and using unix domain sockets fixes
  this problem.
posted: !!timestamp '2023-08-17'
created: !!timestamp '2023-08-17'
time: 12:00 PM
tags:
  - FreeBSD
  - jails
---

The latest iteration of my home firewall has a spare interface.  Many
switches these days have a dedicated management interface, and I wanted
something similar for my firewall.  I want an interface that I can plug in,
get an IP via DHCP, and I can ssh into and it'll "just work" even if
there's a misconfiguration, or an IP conflict.

On FreeBSD,
[vnet jails](https://wiki.freebsd.org/Jails#VNET-based_networking_for_Jails)
are a way to isolate an interface such that it won't interfere (or receive
interference) from other interfaces.  Some may have just used epair and
assigned IPs, but this could cause conflict, while using a unix domain
socket keeps things isolated, and FreeBSD ships w/ everything needed to
make this work.  A package like `socat` isn't needed.

Configuring the host was straight forward, make a directory for the socket:
```
mkdir /var/mgmt
```

And then add a line to `/etc/inetd.conf` to listen to incoming connections
on the socket and launch `sshd`:
```
/var/mgmt/mgmt.ssh.sock stream  unix    nowait  root    /usr/sbin/sshd  sshd -i
```

The `-i` option to `sshd` tells it to run in "`inetd`" mode, which means that
the stdin and stdout of the process is the socket to use for communication.

The next harder part was getting a jail configured that would accept
incoming connections and forward them to the unix domain socket.

First the jail configuration, which goes in `/etc/jail.conf` or similar
location:
```
mgmt {
        host.hostname = mgmt;                           # Hostname

        vnet ="new";
        vnet.interface="mgmt0";

        path = "/usr/jails/mgmt/root";                   # Path to the jail
        mount.fstab="/usr/jails/mgmt/fstab";             # mount spec

        mount.devfs;                                    # Mount devfs inside the jail
        devfs_ruleset = "101";

        exec.start = "/bin/sh /etc/rc";                 # Start command
        exec.stop = "/bin/sh /etc/rc.shutdown";         # Stop command
}
```

There isn't anything special in this.  It's pretty standard jail
configuration, the differences are the vnet configuration, and the
`devfs_ruleset`.

The `devfs_ruleset` is necessary in order to expose the bpf interface
used by dhclient.  This required the following lines in `/etc/devfs.rules`:
```
[mydevfsrules_jail=100]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login

[devfsrules_jail_dhcp=101]
add include $mydevfsrules_jail
add path 'bpf*' unhide
```

Note that after adding the above lines, you need to run:
```
service devfs start
```
to load the rules (per
[devfs(8)](https://man.freebsd.org/cgi/man.cgi?query=devfs&apropos=0&sektion=8&manpath=FreeBSD+13.2-RELEASE+and+Ports&arch=default&format=html)
man page).

Note: I learned my mistake not to number my blocks immediately after
the standard defaults (from `/etc/defaults/devfs.rules`).  I had done that
befure but there's now a conflict, so I skip ahead a bit to get a unique range.

Now I needed to make a number of directories for the jail:
```
mkdir -p /usr/jails/mgmt/root
mkdir -p /usr/jails/mgmt/etc
mkdir -p /usr/jails/mgmt/var/mgmt
mkdir -p /usr/jails/mgmt/tmp
```

I needed to setup the `fstab` for the jail:
```
# Device                Mountpoint      FStype  Options         Dump    Pass#
/                       /usr/jail/mgmt/root     nullfs  ro      0       0
/usr/jail/mgmt/etc      /usr/jail/mgmt/root/etc nullfs  rw      0       0
/usr/jail/mgmt/var      /usr/jail/mgmt/root/var nullfs  rw      0       0
/var/mgmt               /usr/jail/mgmt/root/var/mgmt    nullfs  ro      0       0
/usr/jail/mgmt/tmp      /usr/jail/mgmt/root/tmp nullfs  rw      0       0
```

This is a little bit more tricky, It first `nullfs` mounts the root system.
I'm using ZFS boot environments, so this is a pretty clean FreeBSD install
without much host specific data.  It then mounts some jail specific directories
for `etc`, `tmp` and `var` and finally mounts the shared directory w/ the
unix domain socket to the host system.  Also note that a couple of the mounts
are read-only to prevent the jail from modifying the system.

The `etc` directory was populated from the system `etc` via:
```
tar -cf - -C /etc . | tar -xf - -C /usr/jails/mgmt/etc
```

Then the jail was configured, first `/usr/jails/mgmt/etc/rc.conf`:
```
hostname="mgmt.funkthat.com"

sendmail_enable="NONE"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

# Management port
ifconfig_mgmt0="DHCP"

# necessary as devd can't be run in a jail
synchronous_dhclient="YES"

inetd_enable="YES"              # Run the network daemon dispatcher (YES/NO).
```

The key part of this configuration that took me a while to figure out was
the `synchronous_dhclient` line.  It used to be that `netif` would start
dhclient, but in order to better handle USB ethernet devices and other
removable interfaces, it was moved to `devd`.  The only problem is that
`devd` hasn't been jail'ified, and you can't run it to get things like
link notifications that would normally launch dhclient.  Setting this to
yes, makes sure it gets launched when the jail starts.

And then the following line was added to `/usr/jails/mgmt/etc/inetd.conf`:
```
ssh     stream  tcp     nowait  root    /usr/bin/nc     nc -N -U /var/mgmt/mgmt.ssh.sock
```

This is the part that will forward incoming connections to the ssh port
on to the unix domain socket.

Now that everything is configured, a simple `jail -c mgmt` will get
the jail running and accepting connections.

This was testing and deployed on a FreeBSD 14-CURRENT build as of August
8th, 2023, or more specifically, from `main-n264621-09c20a29328`, but it
should work on all currently supported FreeBSD releases.