diff --git a/README.md b/README.md index f5afc94..255fb7e 100644 --- a/README.md +++ b/README.md @@ -1,28 +1,43 @@ # medashare (Meta Data Sharing) -The idea for medashare is both a standard, but also an implementation of defining and sharing metadata about files. Some media contains the ability to embed metadata, e.g. mp3, some documents, video files, but not all files are able to convey the complete and rich information that may be desired. This will allow you to identify files, and have external assosicated metadata with various files. +The idea for medashare is both a standard, but also an implementation of defining and sharing meta data about files. Some media contains the ability to embed meta data, e.g. mp3, some documents, video files, but not all files are able to convey the complete and rich information that may be desired. This will allow you to identify files, and have external associated meta data with various files. -The idea is also to be able to classify parts of the metadata as private (aka TLP:Red), such that the information will not be shared, so that you can mark files w/ your own tags and/or ratings, which will not be automatically shared. +The idea is also to be able to classify parts of the meta data as private (aka TLP:Red), such that the information will not be shared, so that you can mark files w/ your own tags and/or ratings, which will not be automatically shared. -There will also be able to define derivitive works by the standard. For example, an album may have multiple tracks, so you can note that an mp3 file is a segment of the album, or that the mp3 file obtained from a music service is equivalent to the track. This will allow sharing metadata between mediums. +There will also be able to define derivative works by the standard. For example, an album may have multiple tracks, so you can note that an mp3 file is a segment of the album, or that the mp3 file obtained from a music service is equivalent to the track. This will allow sharing meta data between mediums. -This derivation is also useful for when files are programaticly transformed. Say an image is resized, adding a note that the smaller file is a derivative work can allow others to reproduce the file, but also allow you to not have to reenter all the metadata associated with the new version of the file. +This derivation is also useful for when files are programmatically transformed. Say an image is resized, adding a note that the smaller file is a derivative work can allow others to reproduce the file, but also allow you to not have to reenter all the meta data associated with the new version of the file. This can be useful for things like raw files on a camera, where you associate general picture information w/ the raw image data (but none of the associated processing data that files like CR2 contains) so that the meta data is not lost. -This work is inspired by my work on STIX, a Cyber Threat Inteligence standard, that has many similar requirements as meta data sharing. +This work is inspired by my work on STIX, a Cyber Threat Intelligence standard, that has many similar requirements as meta data sharing. -## Goals +## Goals / Use Cases 1. Provide meta data, such as title, actors, copyright holder, for files, such as movies, photos, documents. 2. Allow look up of meta data by title, actors, etc. 3. Look up meta data belonging to a file, via file hash. -4. Support embedded files, such as within a zip file, or bittorrent, so the querier can get all the meta data for a container file, or that the file can be located for download. -5. Identify transformations of files, such as a reencoding of a movie, or a resizing of a photo. -6. Possily use of fingerprint technology, so that the database can be to query metadata based upon parts of the audio/video/image. -7. Provide metadata for other objects too, such as suggested page down locations for PDFs, or what parts of PDF should be kept on screen in a complete set, so that a page down keeps things readable (and you don't have to arrow up). +4. Support embedded files, such as within a zip file, or bittorrent, so the querier can get all the meta data for a container file, or that the file can be located for download. For example, the info hash for FreeBSD 11.2-R, which then can be d/l'd. +5. Identify transformations of files, such as a reencoding of a movie, resizing of a photo, or clips of audio/video. For example, a CD often has tracks, and there may be a file that is the whole CD, or just one or part of a track. Both directions should be supported, noting a track is part of a album and when an album has tracks, and links to them. +6. Possibly use of fingerprint technology, so that the database can be to query meta data based upon parts of the audio/video/image. +7. Provide meta data for other objects too, such as suggested page down locations for PDFs, or what parts of PDF should be kept on screen in a complete set, so that a page down keeps things readable (and you don't have to arrow up). +8. Links to other repositories, such as YouTube videos, SoundCloud, etc. +9. i18n. Provide translations for fields as needed. Often movie titles will have different translations for different markets/languages. Actors may have different names (e.g. Chinese name vs English name). +10. Overlaying/replacing meta data from someone else's object. This may include deleting properties. Say an actor is missing, or you want to add them to it, or you've encoded the DVD, and you just link to someone's BluRay version. -## MetaData Object +## Types + +Everything must have a type. Not having well defined types can lead to confusion and problems. Different encoding schemes have different ways of encoding types. If the encoding scheme has a native way to encode that type, it should be used. In some cases, e.g. JSON, there is no formal types beyond numbers and strings, and in this case, a type should (MUST? or via schemas?) be layered on top. + +### Integers + +Look at adding units. + +## Objects + +These are the nodes that contain a majority of the data. + +### MetaData Object Properties: uuid UUIDv4 @@ -32,21 +47,81 @@ object_marking_refs Imported from [STIX v2.0 Part 1]: Section 3.1 granular_markings Imported from [STIX v2.0 Part 1]: Section 3.1 Opinion Properties: -qualityrating On a scale from 1 (poor/terrible) to 5 (great/pristine), the subjective quality of the content +qualityrating On a scale from 1 (poor/terrible) to 5 (great/pristine), the subjective quality of the content. The base object will contain all the data associated w/ the file. The base set of data is based upon the [Dublin Core] specification, as it provides a nice starting point, and will provide a good mapping to other systems out there. There may be a link to another MetaData object from which this one is derived. If there is, all the meta data from the derived object (and the ones it derives from) must be included, except for the ones that have been marked deleted, or were overridden. When a property is marked as opinion, it should not be inherited. If the new author agrees with the opinion, then they have to restate the opinion in their object. -Custom properties must be preceeded w/ a namespace. The name space is name followed by colon, as is demonstracted above w/ dc for [Dublin Core]. +Custom properties must be preceded w/ a namespace. The name space is name followed by colon, as is demonstrated above w/ dc for [Dublin Core]. + +The link to the meta data object must include the version referenced, as the referenced object may change. A three way merge may be needed when updating an object where the derived object has also been updated if the new information is wished to be used. + +Open Questions: When meta data is "declassified", how do you maintain a link to the classified version? + +### File Object + +A file object references a MetaData Object, and contains information about the file that the meta data object is associated with. + +### Container Object + +A container object references one or more File objects. This is for representing containers such as zip or tar.gz files, but is also for BitTorrent hashes (event for single file torrents). + +## Links + +These are the edges that connect the nodes. For the most part they do not contain any data. + +[//]: # (Do we for containers? Shouldn't the File be unique, or if not, doesn't mater?) -The link to the metadata object must include the version referenced, as the referenced object may change. A three way merge may be needed when updating an object where the derived object has also been updated if the new information is wished to be used. +### Equivalent -Open Questions: When metadata is "declassified", how do you maintain a link to the classified version? +The two linked nodes, required to be File Objects, are equivalent. -## File Object +## Questions -A link object references a MetaData Object, and contains information about the file that the metadata object is associated with. +# Open + +1. Fully embedded links or have a separate node object? Embedded links have the advantage of being smaller, but require more structure in the parent object. This structure is likely needed in some cases, such as albums w/ tracks, but some edges, such as a clip to a movie still needs to contain data (meaning not so much a node). I'm leaning towards embedded for now, as this should make things easier, and often structure is needed. +2. How to handle similar, but split meta data? One person decides to make a simple meta data object for a scene from a movie, while another person makes a segment of that scene from the movie. Should the segment object be a link between the two? or contain it's own proper data? Some of this can be handled w/ an equivalent meta data object to link two meta datas as being the same. +3. For quality, is this talking about the possible representation, or the actual "content"? So, a VHS, or old analog over the air encoding may be crappy, but the movie content may be good. We may want to do a multi layered approach (this is less than ideal due to complexities), where files can only link to info about that file, i.e. coding, format, resolution, and this meta data object links to one that is the actual content, i.e. movie w/ actors. Or should this be done via overlay? i.e. someone creates a BluRay meta data object about a movie, and then the DVD overlays the DVD resolution and other info, w/ deleting properties that are not relevant. + +# Settled / Likely Closed + +1. Does a track of a CD deserve it's own "meta data" object? Thinking yes, as the track may be played on radio, etc. And the Album object can point to the tracks. This also helps solve the compilation problem as the artists and other details are easier to represent separately. [Dublin Core]: http://dublincore.org/documents/dces/ [STIX v2.0 Part 1]: http://docs.oasis-open.org/cti/stix/v2.0/cs01/part1-stix-core/stix-v2.0-cs01-part1-stix-core.html + +# Some thoughts + +DHT looks like a good option for finding things. IPFS is a great option for storing the data, and allowing peers to find the data, but it does NOT provide a search solution. It should be able to combine the hash tree crypto solution along w/ the DHT to provide a way to build up an index for a peice of data. + +Need to look at DSHT +Thoughts: + +For search, you need two functions: +1. lookup(term) +2. addobject(term, object) + +How to do lookup: +1. Generate a hash of the search term: searchhash = hash(term) +2. Do a query of this hash to find if there is an object at this location, and this hash will reference an object that contains the results. + +How to add an object: +1. Do a lookup and fetch the object that contains all the current objects. +2. Update object w/ new object, and now publish this new object. + + +Validating the object: + +Attacks to prevent: +Adding random hashes that don't map to anything. +Adding valid hashes that don't have the proper term in them. +When adding hashes, limit number of unverified hashes per block iteration. + +Issue is, how do accept that a new block is valid: +Some items are attempted to be fetched (likely based upon generation) and validated. Ones that are not validated are marked as suched, and after a period of time of remaining unvalidated are removed. +New objects likely need to be validated in the immediately following block to help prevent bad growth. +Likely there needs to be multiple "live" blocks that are intermingled. This can be done via a simple LSFR + count, likely dependant upon number of updates and size and difficulty of validating objects. + +When updating, always check for n + 1 until n is not found. When publishing, depending upon timeframe, select n where n is smallest but still means time parameter.