jmg
/
blog


			
				
					
						
						
							
							---
title: Making FreeBSD magnet links
description: >
  Making FreeBSD magnet links
created: !!timestamp '2018-07-03'
time: 4:49 PM
tags:
  - FreeBSD
  - magnet
---

For the last few years, I've been producing torrents and publishing
magnet links, but there is some special work that I do to make these.
The first few releases, I inserted a bogus tracker into the torrent,
because despite there being plenty of tools out there for producing
trackerless (DHT) torrents, they were all GUI and I never found any that
were command line based.  The other was there was/is no tool for
extracting the info hash and building the magnet link.  There may be
tools now, but I couldn't find any when I started 3 years ago.

The following steps are based upon the recent release of FreeBSD 11.2-R,
adjust as necessary.

1. Fetch FreeBSD into a directory (I create a per release directory).  There
   are a few directories that you have mirror, I use wget for this.  The
   mirroring feature for wget isn't great.  After each command I have to
   remove the `CHECKSUM.SHA256`, `CHECKSUM.SHA512` and `index.html*` files.
   ```
   $ wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/ISO-IMAGES/11.2/
   $ wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/VM-IMAGES/11.2-RELEASE/aarch64/Latest/
   $ wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/VM-IMAGES/11.2-RELEASE/amd64/Latest/
   $ wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/VM-IMAGES/11.2-RELEASE/i386/Latest/
   ```
2. Fetch the signature files:
   ```
   $ wget https://www.freebsd.org/releases/11.2R/CHECKSUM.SHA512-FreeBSD-11.2-RELEASE-{amd64,i386,powerpc,powerpc-powerpc64,sparc64,arm64-aarch64}.asc
   $ wget https://www.freebsd.org/releases/11.2R/CHECKSUM.SHA512-FreeBSD-11.2-RELEASE-{amd64,i386,arm64-aarch64}-vm.asc
   $ wget https://www.freebsd.org/releases/11.2R/CHECKSUM.SHA512-FreeBSD-11.2-RELEASE-arm-armv6-{BANANAPI,BEAGLEBONE,CUBIEBOARD,CUBIEBOARD2,CUBBOX-HUMMINGBOARD,GUMSTIX,PANDABOARD,RPI-B,RPI2,WANDBOARD}.asc
   ```
3. Verify the GPG key that signed the above files.  This is usually Glen
   Barber's key, but not always.  I have met and verified his fingerprint
   in person,  If you have verified someone's key who has signed Glen's
   key, that is another good way.
4. Verify the checksum files:
   ```
   $ for i in *.asc; do gpg --verify $i; done
   You should see a bunch of lines like:
   Warning: using insecure memory!
   gpg: Signature made Fri Jun 22 09:33:50 2018 PDT
   gpg:                using RSA key 0x031458A5478FE293
   gpg: Good signature from "Glen Barber <gjb@FreeBSD.org>" [full]
   gpg:                 aka "Glen Barber <glen.j.barber@gmail.com>" [full]
   gpg:                 aka "Glen Barber <gjb@glenbarber.us>" [full]
   gpg:                 aka "Glen Barber <gjb@keybase.io>" [unknown]
   gpg: WARNING: not a detached signature; file 'CHECKSUM.SHA512-FreeBSD-11.2-RELEASE-amd64-vm' was NOT verified!
   ```
   The last line can be ignored.  The non-`.asc` files were d/l'd and will
   not be used.  Make sure that all of the files report Good signature.
5. In the past I have used BitTornado for other things, so I ended up
   using it as the basis to make the tool for creating trackerless torrent
   files.  The modifications were simple.  It appears that the original
   BitTornado CVS tree is off-line (anyways, it was served insecurely),
   but it looks like
   [effigies/BitTornado](https://github.com/effigies/BitTornado) is
   similar enough that it could be modified and used.  I copied
   `btmakemetafile.py` to `btmaketrackerless.py` and applied the following
   patch:
   ```
   $ diff -u btmakemetafile.py btmaketrackerless.py 
   --- btmakemetafile.py   2004-05-24 12:54:52.000000000 -0700
   +++ btmaketrackerless.py        2016-10-10 17:13:32.742081000 -0700
   @@ -23,9 +23,9 @@
    def prog(amount):
        print '%.1f%% complete\r' % (amount * 100),

   -if len(argv) < 3:
   +if len(argv) < 2:
        a,b = split(argv[0])
   -    print 'Usage: ' + b + ' <trackerurl> <file> [file...] [params...]'
   +    print 'Usage: ' + b + ' <file> [file...] [params...]'
        print
        print formatDefinitions(defaults, 80)
        print_announcelist_details()
   @@ -33,9 +33,9 @@
        exit(2)

    try:
   -    config, args = parseargs(argv[1:], defaults, 2, None)
   -    for file in args[1:]:
   -        make_meta_file(file, args[0], config, progress = prog)
   +    config, args = parseargs(argv[1:], defaults, 1, None)
   +    for file in args[0:]:
   +        make_meta_file(file, None, config, progress = prog)
    except ValueError, e:
        print 'error: ' + str(e)
        print 'run with no args for parameter explanations'

   ```
   If you notice, the only thing that is done is to drop the first argument,
   and instead of passing it into `make_meta_file`, a `None` is passed
   instead.  This will simply not add trackers to the torrent file.
6. I then run the following script to verify the downloaded files, and
   generate the torrent files:
   ```
   $ cat cmp.sh 
   #!/bin/sh -
   # wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/ISO-IMAGES/11.2/
   # wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/VM-IMAGES/11.2-RELEASE/aarch64/Latest/
   # wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/VM-IMAGES/11.2-RELEASE/amd64/Latest/
   # wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/VM-IMAGES/11.2-RELEASE/i386/Latest/
   # wget https://www.freebsd.org/releases/11.2R/CHECKSUM.SHA512-FreeBSD-11.2-RELEASE-{amd64,i386,powerpc,powerpc-powerpc64,sparc64,arm64-aarch64}.asc
   # wget https://www.freebsd.org/releases/11.2R/CHECKSUM.SHA512-FreeBSD-11.2-RELEASE-{amd64,i386,arm64-aarch64}-vm.asc
   # wget https://www.freebsd.org/releases/11.2R/CHECKSUM.SHA512-FreeBSD-11.2-RELEASE-arm-armv6-{BANANAPI,BEAGLEBONE,CUBIEBOARD,CUBIEBOARD2,CUBBOX-HUMMINGBOARD,GUMSTIX,PANDABOARD,RPI-B,RPI2,WANDBOARD}.asc

   grep -h '^SHA512' CHECK*.asc | sed -e 's/SHA512 (\(.*\)) = \(.*\)/\2 \1/' | sort -k 2 > sha512.from.asc

   while read hash fname; do
           if [ -e "$fname" ]; then
                   sigfile=`grep -l -- "$fname" *.asc | head -n 1`
                   echo checking "$fname", sig in: "$sigfile"
                   #res=`sha512 -q "$fname"`
                   res=`shasum -a 512 "$fname" | awk '{ print $1 }'`
                   echo "File is: $res"
                   if [ x"$res" != x"$hash" ]; then
                           echo missmatch!  "$fname"
                           exit 1
                   fi
                   if ! [ -e "$fname".torrent ]; then
                           btmaketrackerless.py "$fname"
                   fi
           else
                   echo missing "$fname"
                   exit 1
           fi
   done < sha512.from.asc
   ```
7. Once all the torrents have been generated, I then make the magnet
   links:
   ```
   $ cat btmakemagnet.sh 
   #!/bin/sh -

   # metainfo file.: FreeBSD-10.3-RELEASE-sparc64-bootonly.iso.torrent
   # info hash.....: 06091dabce1296d11d1758ffd071e7109a92934f
   # file name.....: FreeBSD-10.3-RELEASE-sparc64-bootonly.iso
   # file size.....: 203161600 (775 * 262144 + 0)
   # announce url..: udp://tracker.openbittorrent.com:80
   # btshowmetainfo 20030621 - decode BitTorrent metainfo files

   for i in *.torrent; do
           btshowmetainfo.py "$i" | awk '
   $0 ~ "^info hash" { info = $3 }
   $0 ~ "^file name" { name = $3 }
   END {
           print "magnet:?xt=urn:btih:" info "&dn=" name
   }'
   done
   ```
8. I then create the magnet links file, and update the
   [Torrents](https://wiki.freebsd.org/Torrents) wiki page.

Sorry about the code formatting.  I don't know how to make it look better
in blogger.