jmg
/
fbsdembdev
1
0
Fork 0
Code Issues 0 Pull Requests 0 Releases 0 Wiki Activity
Browse Source

try to use consistent names/terms, and update to use the new terms 

picked.  Make the diagram generic and use the new terms...
main
John-Mark Gurney 4 years ago
parent
ba0e55e819
commit
86cd092139
2 changed files with 156 additions and 120 deletions
Split View
Diff Options
Show Stats Download Patch File Download Diff File
  1. +8
    -6
      diag.getxt
  2. +148
    -114
      lab.doc.md.html

+ 8
- 6
diag.getxt View File

@@ -1,8 +1,10 @@
[ Internet ] { flow: north } -- [ PoE/VLAN Switch ]
[ Host machine ] -- [ PoE/VLAN Switch ]
[ Host machine ] { flow: south } -- [ USB Hub ]
[ Internet ] { origin: PoE/VLAN Switch; offset: 0, -2 } -- [ PoE/VLAN Switch ]
[ Controller ] -- { end: west } [ PoE/VLAN Switch ]
[ Controller ] { flow: south } -- [ USB Hub ]
[ PoE Splitter ] { flow: north } -- [ PoE/VLAN Switch ]
[ RockPro64 ] -- { label: "Network" } [ PoE Splitter ]
[ RockPro64 ] -- { label: "Power" } [ PoE Splitter ]
[ Board ] -- { label: "Network" } [ PoE Splitter ]
[ Board ] -- { label: "Power" } [ PoE Splitter ]
[ USB Hub ] -- [ Serial adapter ]
[ Serial adapter ] { flow: south; } -- [ RockPro64 ]
[ USB Hub ] { flow: north } -- [ SDWire ] { origin: USB Hub; offset: 0, 2 }
[ SDWire ] -- [ Board ]
[ Serial adapter ] { flow: south; } -- [ Board ]

+ 148
- 114
lab.doc.md.html View File

@@ -1,25 +1,40 @@
<meta charset="utf-8">
<meta http-equiv="refresh" content="2">

Embedded Board dev cluster
==========================
**Embedded Board Dev Lab**

This is the description of how the cluster is architected and setup. There
Overview
========

This is the description of how the lab is architected and setup. There
are a few design decisions that are likely to be different in other
environments, but this made the most sense for mine.
environments, but hopefully the overall architecture will be similar.
The differences should be laid the documentation for their instances.

Goal
----

The goal of this setup is to allow people to be able to install and test
images on various boards remotely.

### Definitions

- User - A person or entity that is able to reserve a system, and test with it.
- Host system - The machine that controls and provides access to the systems.

### Features
The goal of this setup is to allow users to be able to install and test
FreeBSD patches and images on boards that they do not have locally.

Definitions
-----------

- Board - The system that is available for testing. This can be an SBC,
another computer, or any other system that needs to be tested.
- Board class - A set of boards of similar type.
- Board Handle - An identifier for a specific board instance. It will be
consistent until the lab is reconfigured. It can be used to help debug
issues that affect a specific board, or reserve a specific board that
has special hardware, such as an SDWire attached to it.
- Controller - The machine that controls and grants access to the boards.
- Jail - A jail on the controller that the User is able to log into. It
has access to the board, such as serial and ethernet. It will also be
able to power cycle the board.
- User - A person or entity that is able to reserve a board, and use it.

Features
--------

- Remote power control
- serial console access
@@ -30,135 +45,154 @@ images on various boards remotely.
- Long term goals:
- Emulated microSD cards

### Physical Architecture

The following diagrams the connections between the components. It is expected
that the connections for the RockPro64 is followed similarly for other embedded
devices.

In the case of the Switch, the RockPro64 will be put on a VLAN which will be
delivered to the Host machine tagged. This will allow a jail in the host
machine to have a direct control over the broadcast domain for the device.
This will allow running dhcp/bootp/tftp services for netbooting by running dnsmasq
or another service. As fusefs is now jail friendly, the root FS could even be
mounted via sshfs

The PoE part of the switch will be used for power control. PoE splitters (~$10)
are readily available and inexpensive, and the cost per port is realatively
inexpensive considering that power consumption will also be provided.

**************************************************************************
* *
* +--------------+ +-----------------+ *
* | Host machine +-------------------------+ | Internet | *
* +-+------------+ | +-+---------------+ *
* | | | *
* | | | *
* | | | *
* +-+------------+ +----------------+ | +-+---------------+ *
* | USB Hub +-----+ Serial adapter | +---------+ PoE/VLAN Switch | *
* +--------------+ +-+--------------+ +-+---------------+ *
* | | *
* | | *
* | | *
* +-+--------------+ Network +-+---------------+ *
* | RockPro64 +------------+ PoE Splitter | *
* +-+--------------+ +-+---------------+ *
* | Power | *
* +-----------------------------+ *
* *
**************************************************************************

### Logical Architecture

No user will be able to log into the host machine directly. The only user
interface exposed on the host will be via an RPC interface, e.g.
`echo function xxx | ssh labhost labcli`, or via a socket from within
the jail.

The user will be able to log into the jail that is created during the
reservation. Any modifications to the jail will be rolled back and
discarded after the system has been released, or the reservation has
expired.
Physical Architecture
---------------------

The following diagrams the connections between the components. There will
be many different instances of the board.

The switch will have a unique VLAN assigned to each board. The controller
will receive the VLANs tag to deliver them to the correct jail. This
allows a jail on the host machine to have a direct control over the
broadcast domain for the device. It will allow running dhcp/bootp/tftp
services for netbooting. As fusefs is jail friendly, the root FS could
even be mounted via sshfs and exported to the board via NFS.

The PoE part of the switch will be used for power control. PoE splitters
(~$10) are readily available and inexpensive, and the cost per port is
realatively inexpensive. The switch will be able to provide granular power
consumption on a per board basis.

Some instances of the board will use an
[SDWire](https://wiki.tizen.org/SDWire). The SD reader will be accessible
in the jail, and a mechanism will be provided to switch the microSD card
between the jail and the board. This will allow easier testing of images.

************************************************************************
* *
* +------------+ +-----------------+ *
* | Controller +-------------------------+ | Internet | *
* +-+----------+ | +-+---------------+ *
* | | | *
* | | | *
* | | | *
* +-+----------+ +----------------+ | +-+---------------+ *
* | USB Hub +-----+ Serial adapter | +---------+ PoE/VLAN Switch | *
* +-+----------+ +-+--------------+ +-+---------------+ *
* | | | *
* | | | *
* | | | *
* +-+----------+ +-+--------------+ Network +-+---------------+ *
* | SDWire +-----+ Board +------------+ PoE Splitter | *
* +------------+ +-+--------------+ +-+---------------+ *
* | Power | *
* +-----------------------------+ *
* *
************************************************************************

Logical Architecture
--------------------

No user will be able to log into the controller directly. The only
interface exposed to the user on the controller will be via an RPC
interface, e.g. `echo function xxx | ssh labhost labcli`, or via a control
socket from within the jail.

The user will be able to log into the jail that is created for the
reservation of a board. Any modifications to the jail will be rolled
back and discarded after the system has been released, or the reservation
has expired. A persistant user directory will be mounted and shared
between all jails, i.e. if the user has three different boards reserved,
each jail will have a nullfs mount of the user's directory.

Workflow
--------
========

# Functions
Functions
---------

The functions that take a device handle can be executed from within the device jail
and the device handle of that device jail will be used. An error will be raised if
a device handle is provided and it does not match the current jail.
The functions that take a board handle argument can be executed from within
the jail and the board handle of that jail will be used. The board handle
argument will be ignored and is optional when used from withing the jail.

These are the functions a user can execute:
1. List device classes (onlyavailable=false)
List the device classes that are known about. If onlyavailable is true, than only
ones that are currently available for claiming are returned.
2. Device status (claimed=false)
List all the device statuses currently. This includes that status, claimed or
unclaimed. If the device is claimed, it includes the user to claimed it. If claimed
is true, only list devices that are currently claimed by you. The default is to list
all devices.
3. Claim device (device class, power=false)
A user can use this to lock a device. This will return a device handle with the
device information, such as the device's jail's IP address, or an error. Once they
have obtained a device, it will not be allocated to another user till they have release
this device (or in the future a timeout has been hit). The user's ssh keys will be
automatically populated in that jail. By default, the device will be in the power off
state. When a default configuration is provided for the board, it can be automatically
powered on to make it easier to integrate w/ systems like CI.
4. Reinit device (device handle)
This will remove the current jail, and recreate it as if it was claimed for the first
time. This can be done so you can get the jail in a clean state w/o risk losing the
lock by freeing it and then reclaiming it. If you have not claimed the device, an
error will be returned. If the reinit fails, an error will be returned, but the claim
will be maintained.
5. Release device (device handle)
Release a claim on the device handle returned by claim device. This will make it
available to users again. All data in the jail will be deleted. It will return an
error if you do have have a claim on the device.
6. Power off (device handle)
Turn off the power to the device.
7. Power on (device handle)
Turn on the power to the device.

# Services provided in the device jail
1. List board classes (onlyavailable=false)
List the board classes that are on the controller. If onlyavailable is
true, than only the classes that have boards available for reservation
are returned.
2. Board status (reserved=false)
List all the board statuses. This includes the status, reserved or
unreserved. If the board is reserved, it includes the user who reserved
it. If reserved is true, only list boards that are currently reserved
by you. The default is to list all boards.
3. Reserve board (board class or board id, power=false)
A user can use this to reserve a board. This will return a board handle
with the board information, such as the board's jail's IP address, or an
error. Once the user has obtained a board, it will not be allocated to
another user till they have release this board (or a timeout has
expired). The user's ssh keys will be automatically populated in the
jail. By default, the board will be in the power off state. When a
default configuration is provided for the board, it can be automatically
powered on, via the power arugment being true to make it easier to
integrate w/ systems like CI.
It is expected that in the future, additional arguments will allow the
user to specify different environments (e.g. stable/12).
4. Reinit board (board handle)
This will kill all processes and remove the current jail. The jail will
be recreated as if it was reserved for the first time. This allows the
jail to be in a clean state w/o losing the reservation by releasing it
and then reclaiming it. This will be important if there are pending
reservations for the board. If the user has not claimed the board, an
error will be returned. If the reinit fails, an error will be returned,
but the claim will be maintained if possible.
5. Release board (board handle)
Release a reservation on the board handle returned by reserve board
function. This will make the board available to users again. All data
in the jail will be deleted. The function will return an error if the
user does not have a claim on the board.
6. Power off (board handle)
Turn off the power to the board.
7. Power on (board handle)
Turn on the power to the board.

Services provided in the jail
-----------------------------

Once logged in, the jail will have the following services:
1. ssh
Incoming ssh must be provided for the user to login.
2. One interface for internet
nat setup so that all traffic appears from jail's IP.
3. One interface for device
3. One interface for board
4. Serial port for console access
The configuration will identify which device to put in the jail. For
The configuration will identify which board to put in the jail. For
USB devices, they will be able to be specified via a list of ports on
the hubs so that there is no issue w/ device probe order.
5. dhcpd for device w/ tftp already configured
5. dhcpd for boards w/ tftp already configured
6. inetd w/ tftpd setup
7. nfsd setup w/ exports configured
8. power control for device
8. power control for board

As the user will have root access, all of these can be modified after.
As the user will have root access, all of these can be modified.

Future Work
===========

There are a number of ideas to make developing boards remotely more doable.
We can use devices that support OTG, to be USB devices for test boards. This
means you can simulate a keyboard and or a mouse. With the combination of a
HDMI to ethernet adapter (there are cheap ones for ~$40/each), a developer can
work on making X or other GUI work remotely.
We can use devices that support OTG, to emulate USB devices for test boards.
This would allow the simulate a keyboard and or a mouse. With the addition
of an HDMI to ethernet adapter (there are cheap ones for ~$40/each), a user
can work on making X or other GUI work remotely.

Questions
=========
Open Questions
==============

Should the host key for the device be kept between invocations.
Pros: Users don't have to munch the known_hosts every time.
Should the jail's host key for the board be kept between invocations.
Pros: Users don't have to modify the known_hosts every time.
Cons: Malicious user could impersonate the jail as they can copy out the key.

IPv6 support?
I have enough that each device can get a /64, but this remove privacy in
that the network will tell which device was active at the time
that the network will tell which board was active at the time

<!-- Markdeep: --><style class="fallback">body{visibility:hidden;white-space:pre;font-family:monospace}</style><script src="markdeep.min.js" charset="utf-8"></script><script src="https://casual-effects.com/markdeep/latest/markdeep.min.js" charset="utf-8"></script><script>window.alreadyProcessedMarkdeep||(document.body.style.visibility="visible")</script>

Write Preview
Loading…
Cancel
Save