diff --git a/NOTES.txt b/NOTES.txt index 42f0748..00c79e8 100644 --- a/NOTES.txt +++ b/NOTES.txt @@ -1,6 +1,11 @@ Fetch all of the archive: scp -r freefall:/local/mail/archive/*/freebsd-snapshots . +Expand all the "validly" formed messages to a file: +zcat $(zgrep -l 'PGP SIGNED MESSAGE' *) > ../complete.txt + +We only lose 2012, and we'll loose everything before 2015 later. + XXX - don't use the following, doesn't decode body properly Explode the mbox to files: cat 201810* freebsd-snapshots | split -p '^From ' - snap. @@ -9,12 +14,11 @@ The stdin trick is needed so the numbering/lettering continues properly. Correct way to split the bodies off. python splitbody.py arch/complete.txt arch/snap. -Expand all the "validly" formed messages to a file: -zcat $(zgrep -l 'PGP SIGNED MESSAGE' *) > ../complete.txt - -We only lose 2012, so no big deal. +Bulk import, -m is used to not check if they exist till the end: +for i in arch/snap.????; do sh addinfo.sh -m $i; done -Bulk import: -for i in snap.??; do sh addinfo.sh -m < $i; done +Clean things up and skip checking for ones before given date +sh addinfo.sh -c 20180900 --m is used to not check if they exist till the end. +Copy them up to freefall: +scp snapshot.* freefall:public_html/FreeBSD-snap diff --git a/README.md b/README.md index f1d0685..1138706 100644 --- a/README.md +++ b/README.md @@ -9,6 +9,27 @@ file. The only file needed for this is the snapaid.sh script. The other files are used for generating the index. +NOTE: The xz vs non-xz versions of some of the images are not able to be +differentiated. Currently shorting rules should always put the xz version +before the non-xz version. + +NOTE: Only snapshots that have SHA512 hashes are included. This excludes +most snapshots from 2015 and before. The tool could be updated to include +SHA256, but not a priority currently. + NOTE: Not all of the snapshots are in the database. Some snapshot names, like 11.0-RC1, don't contain all the info others do, and are not included. In the future, hopefully this will be fixed. + +backend +------- + +The backend is just a simple text file the indexes all the published +snapshots. It is built from the emails to the freebsd-snapshot +list. After verification of the email's signature, the SHA512 entry +lines are extracted, the file name is parsed, and added to the complete +index. The message-id of the email is in the index so that the frontend +can d/l the original email, verify the GPG signature locally. The +complete index is used for verifying a snapshot that has already been +downloaded. Another index is also maintained which only contains the +currently available to d/l snapshots. diff --git a/addinfo.sh b/addinfo.sh index cb223d7..976a857 100644 --- a/addinfo.sh +++ b/addinfo.sh @@ -2,14 +2,28 @@ set -e -args=`getopt m $*` +usage() { + echo 'Usage: $0 [ -m ] ' + echo 'Usage: $0 -c ' + echo '' + echo 'date is specified as YYYYMMDD' + + if [ x"$1" != x"" ]; then + exit $1 + fi +} + +args=`getopt cm $*` if [ $? -ne 0 ]; then - echo 'Usage: $0 [ -m ]' - exit 2 + usage 2 fi set -- $args while :; do case "$1" in + -c) + complete=1 + shift + ;; -m) more=1 shift @@ -19,9 +33,31 @@ while :; do ;; esac done +if [ x"$complete" = x"1" -a x"$more" = x"1" ]; then + echo '-m and -c cannot be specified at the same time.' + usage 2 +elif [ x"$complete" = x"1" -a $# -ne 1 ]; then + echo 'must only specify a date with -c' + usage 2 +elif [ x"$complete" != x"1" -a $# -ne 1 ]; then + echo 'must specify exactly one file' + usage 2 +fi mkdir "$0.running" +if [ x"$complete" = x"1" ]; then + sort -u snapshot.complete.idx | xz > snapshot.complete.idx.xz + awk '$5 >= "'"$1"'" { + if (!system("wget --method=HEAD " $9)) + print + } + ' snapshot.idx | sort -u | xz > snapshot.idx.xz + rm snapshot.idx snapshot.complete.idx + rmdir "$0.running" + exit 0 +fi + # minimize file tmpfname="tmp.snapinf.asc" awk ' @@ -40,7 +76,7 @@ awk ' $0 == "-----END PGP SIGNATURE-----" { output = 0 - }' > "$tmpfname" + }' "$1" > "$tmpfname" if ! gpg --verify "$tmpfname"; then echo 'failed verify'