when I switched to the included enum, the unknown enum exception switched
from KeyError to ValueError, this now fixes that..

fixing strings let me drop some special handlers..

Use BytesIO instead of fileoff so less code to maintain..  Maybe
want to just use bytes instead?

also minor code coverage tweak due to a bug

This still needs some cleanup and additional tests..  This isn't
hooked into the testing system yet as I still haven't decided if
I'm going to commit fixtures or not (or maybe make this it's own
repo)..

IFD needs serious cleanup.. I should be using a classmethod instead
of the janky nextptr bs.

this makes pre-caching files easier, and likely more what a user
expects..

This imports magic.py from file-magic and merges magic_wrap.py into
it...

This also updates detect_from_filename to try w/ _COMPRESS, and if
it returns an error, normal mode.  This is necessary as [some?] zip
files can be decompressed by gzip, but throws an error...

The original query applied a complicated test, which sqlite couldn't
tell if it applied to all..

In the case of any inclusion, it's easy, only search metadata, and match
to files.

If all exclusion, make two parts, the part w/ a metadata object that
doesn't have the exclusions, and the part w/o any metadata objects..

Both of these later two queries can be satified more simply and with
proper indices..

The old query might have worked fine on a more advanced DB, but was
necessary for decent performance..

dump improvements need tests..

Some queries weren't using the full power of SQL and doing some of
the selection in the code, fix that (by_hash and host mapping).

This will help keep popular tags near the top, while expiring less
used tags

My testing machine has 10 cpus, and so didn't trigger the failure
where not all the work was submitted..  We need to pop the completed
work items, and keep doing the for loop while we have futures to
process...  this submits and processes all work..

as we might have a lot of work to submit, BUT it might fail early,
don't submit too much work early on, just for it to fail, so we
limit how much work we submit..

This will be used to allow parallel processing of torrent pieces..

each piece of the torrent can be processed in parallel, and this
class will make sure that when processing the hash of a file in
the torrent, it will be hashed in the correct order...

setdefault works best for simple data types, calling the factory
function every time doesn't make sense...

hide coverage in test code