**Embedded Board Dev Lab** Overview ======== This is the description of how the lab is architected and setup. There are a few design decisions that are likely to be different in other environments, but hopefully the overall architecture will be similar. The differences should be laid the documentation for their instances. Goal ---- The goal of this setup is to allow users to be able to install and test FreeBSD patches and images on boards that they do not have locally. Definitions ----------- - Board - The system that is available for testing. This can be an SBC, another computer, or any other system that needs to be tested. - Board class - A set of boards of similar type. - Board Handle - An identifier for a specific board instance. It will be consistent until the lab is reconfigured. It can be used to help debug issues that affect a specific board, or reserve a specific board that has special hardware, such as an SDWire attached to it. - Controller - The machine that controls and grants access to the boards. - Jail - A jail on the controller that the User is able to log into. It has access to the board, such as serial and ethernet. It will also be able to power cycle the board. - User - A person or entity that is able to reserve a board, and use it. Features -------- - Remote power control - serial console access - Network boot - Internet connectivity for boards w/ ethernet - Isolation between board environments Likely implemented via VLANs+jails w/ VNET to provide complete control - Long term goals: - Emulated microSD cards Physical Architecture --------------------- The following diagrams the connections between the components. There will be many different instances of the board. The switch will have a unique VLAN assigned to each board. The controller will receive the VLANs tag to deliver them to the correct jail. This allows a jail on the host machine to have a direct control over the broadcast domain for the device. It will allow running dhcp/bootp/tftp services for netbooting. As fusefs is jail friendly, the root FS could even be mounted via sshfs and exported to the board via NFS. The PoE part of the switch will be used for power control. PoE splitters (~$10) are readily available and inexpensive, and the cost per port is realatively inexpensive. The switch will be able to provide granular power consumption on a per board basis. Some instances of the board will use an [SDWire](https://wiki.tizen.org/SDWire). The SD reader will be accessible in the jail, and a mechanism will be provided to switch the microSD card between the jail and the board. This will allow easier testing of images. ************************************************************************ * * * +------------+ +-----------------+ * * | Controller +-------------------------+ | Internet | * * +-+----------+ | +-+---------------+ * * | | | * * | | | * * | | | * * +-+----------+ +----------------+ | +-+---------------+ * * | USB Hub +-----+ Serial adapter | +---------+ PoE/VLAN Switch | * * +-+----------+ +-+--------------+ +-+---------------+ * * | | | * * | | | * * | | | * * +-+----------+ +-+--------------+ Network +-+---------------+ * * | SDWire +-----+ Board +------------+ PoE Splitter | * * +------------+ +-+--------------+ +-+---------------+ * * | Power | * * +-----------------------------+ * * * ************************************************************************ Logical Architecture -------------------- No user will be able to log into the controller directly. The only interface exposed to the user on the controller will be via an RPC interface, e.g. `echo function xxx | ssh labhost labcli`, or via a control socket from within the jail. The user will be able to log into the jail that is created for the reservation of a board. Any modifications to the jail will be rolled back and discarded after the system has been released, or the reservation has expired. A persistant user directory will be mounted and shared between all jails, i.e. if the user has three different boards reserved, each jail will have a nullfs mount of the user's directory. Workflow ======== Functions --------- The functions that take a board handle argument can be executed from within the jail and the board handle of that jail will be used. The board handle argument will be ignored and is optional when used from withing the jail. These are the functions a user can execute: 1. List board classes (onlyavailable=false) List the board classes that are on the controller. If onlyavailable is true, than only the classes that have boards available for reservation are returned. 2. Board status (reserved=false) List all the board statuses. This includes the status, reserved or unreserved. If the board is reserved, it includes the user who reserved it. If reserved is true, only list boards that are currently reserved by you. The default is to list all boards. 3. Reserve board (board class or board id, power=false) A user can use this to reserve a board. This will return a board handle with the board information, such as the board's jail's IP address, or an error. Once the user has obtained a board, it will not be allocated to another user till they have release this board (or a timeout has expired). The user's ssh keys will be automatically populated in the jail. By default, the board will be in the power off state. When a default configuration is provided for the board, it can be automatically powered on, via the power arugment being true to make it easier to integrate w/ systems like CI. It is expected that in the future, additional arguments will allow the user to specify different environments (e.g. stable/12). 4. Reinit board (board handle) This will kill all processes and remove the current jail. The jail will be recreated as if it was reserved for the first time. This allows the jail to be in a clean state w/o losing the reservation by releasing it and then reclaiming it. This will be important if there are pending reservations for the board. If the user has not claimed the board, an error will be returned. If the reinit fails, an error will be returned, but the claim will be maintained if possible. 5. Release board (board handle) Release a reservation on the board handle returned by reserve board function. This will make the board available to users again. All data in the jail will be deleted. The function will return an error if the user does not have a claim on the board. 6. Power off (board handle) Turn off the power to the board. 7. Power on (board handle) Turn on the power to the board. Services provided in the jail ----------------------------- Once logged in, the jail will have the following services: 1. ssh Incoming ssh must be provided for the user to login. 2. One interface for internet nat setup so that all traffic appears from jail's IP. 3. One interface for board 4. Serial port for console access The configuration will identify which board to put in the jail. For USB devices, they will be able to be specified via a list of ports on the hubs so that there is no issue w/ device probe order. 5. dhcpd for boards w/ tftp already configured 6. inetd w/ tftpd setup 7. nfsd setup w/ exports configured 8. power control for board As the user will have root access, all of these can be modified. Future Work =========== There are a number of ideas to make developing boards remotely more doable. We can use devices that support OTG, to emulate USB devices for test boards. This would allow the simulate a keyboard and or a mouse. With the addition of an HDMI to ethernet adapter (there are cheap ones for ~$40/each), a user can work on making X or other GUI work remotely. Open Questions ============== Should the jail's host key for the board be kept between invocations. Pros: Users don't have to modify the known_hosts every time. Cons: Malicious user could impersonate the jail as they can copy out the key. IPv6 support? I have enough that each device can get a /64, but this remove privacy in that the network will tell which board was active at the time