fixing the test takes a bit of work, but likely will dump these
older migrations soon..

SQLite3 does not do joins in a sane manner, so we emulate them
w/ subqueries for a large boost.  Not sure if adding distinct would
improve things or not, the query plan does not change between the
two (but the lower ops may), but in a quick test, it didn't seem
to make a difference (not evaluated statistically)...

During attempts to optimize the query, these cases failed, but were
not detected...

not sure how I want to handle spaces or escaping other characters. for now,
don't

second is auto was applying to all files if multiple were specified
on the command line resulting in all the mimetypes applied to all
the files...

just a warning, I've completely forgotten how this metaclass stuff
works...  But it still behaves as it should.

add a class to emulate a file, and only store the part of the file
that was read/accessed...  This reduces storing an 11MB file down
to under 100KB...  It also allows tests to run w/o the whole file...
Put the original files in fixtures/original...

fix up a couple of issues w/ parsing CRW files, and also allow the
ability to skip parts of the CRW file...  This allows skipping
large parts, like the CCD data and the large thumbnail..

when I switched to the included enum, the unknown enum exception switched
from KeyError to ValueError, this now fixes that..

fixing strings let me drop some special handlers..

Use BytesIO instead of fileoff so less code to maintain..  Maybe
want to just use bytes instead?

also minor code coverage tweak due to a bug

This still needs some cleanup and additional tests..  This isn't
hooked into the testing system yet as I still haven't decided if
I'm going to commit fixtures or not (or maybe make this it's own
repo)..

IFD needs serious cleanup.. I should be using a classmethod instead
of the janky nextptr bs.

this makes pre-caching files easier, and likely more what a user
expects..

This imports magic.py from file-magic and merges magic_wrap.py into
it...

This also updates detect_from_filename to try w/ _COMPRESS, and if
it returns an error, normal mode.  This is necessary as [some?] zip
files can be decompressed by gzip, but throws an error...

The original query applied a complicated test, which sqlite couldn't
tell if it applied to all..

In the case of any inclusion, it's easy, only search metadata, and match
to files.

If all exclusion, make two parts, the part w/ a metadata object that
doesn't have the exclusions, and the part w/o any metadata objects..

Both of these later two queries can be satified more simply and with
proper indices..

The old query might have worked fine on a more advanced DB, but was
necessary for decent performance..

dump improvements need tests..

Some queries weren't using the full power of SQL and doing some of
the selection in the code, fix that (by_hash and host mapping).

This will help keep popular tags near the top, while expiring less
used tags

My testing machine has 10 cpus, and so didn't trigger the failure
where not all the work was submitted..  We need to pop the completed
work items, and keep doing the for loop while we have futures to
process...  this submits and processes all work..