WWDC2014 Session 505

Transcript

[ Silence ] >> Hello, good afternoon, welcome to session 505, [ Silence ] "Harnessing Metadata in Audiovisual Media". [ Silence ] I've heard we are competing with the "Intro to Swift" talk [ Silence ] so get intimate with your neighbors here. [ Silence ] My name is Adam Sonnanstine. [ Silence ] I'm an engineer on the AVFoundation team [ Silence ] and today we are going to talk about, of course, metadata. [ Silence ] So what do I mean by metadata? [ Silence ] Well, for the purposes of this talk we're going [ Silence ] to define metadata to mean any data that is stored [ Silence ] in movie files, streaming presentations, any other sort [ Silence ] of audiovisual presentation that describes the primary data, [ Silence ] like the audio and video that we think about when we think [ Silence ] of those sorts of presentations. [ Silence ] Some examples are always helpful. [ Silence ] One you should be familiar is iTunes metadata, [ Silence ] when you have this sort of the song names and the artists [ Silence ] and the album artwork in your iTunes library. [ Silence ] All these things are stored as the sort of metadata [ Silence ] that I'm talking about today in the files [ Silence ] in your iTunes library. [ Silence ] Besides iTunes metadata, we also have things [ Silence ] like location information. [ Silence ] If you have a movie that you took with your iPhone [ Silence ] or some other location-enabled device and you played it [ Silence ] in QuickTime player, that will show up in the info window [ Silence ] to tell you where you were when you took that movie. [ Silence ] That's also stored as the kind of metadata [ Silence ] that we are talking about today. [ Silence ] Some new features, we know that you're not always standing still [ Silence ] when you are taking your videos. [ Silence ] So new in iOS 8 and OS X Yosemite, we have features [ Silence ] that support things like dynamic location, that's a location [ Silence ] that changes over time. [ Silence ] So these are some new features that we are going to be talking [ Silence ] about later on that we are pretty excited about and, [ Silence ] in addition to location, this really applies to any sort [ Silence ] of metadata that you might want to add that changes [ Silence ] over time in your movie. [ Silence ] This is a screen shot of a demo app we'll show you later [ Silence ] but the circle and the annotation text that's all [ Silence ] stored as the same sort of timed metadata as a timed location. [ Silence ] So hopefully that whets your appetite a little bit. [ Silence ] We'll talk about what we're going to cover today, [ Silence ] we're going to start, I'm going to give you an intro to metadata [ Silence ] and AVFoundation, some of the classes that have been [ Silence ] around for a while for describing all sorts [ Silence ] of metadata, how to inspect that and how to author it. [ Silence ] We're going to talk more [ Silence ] about those new timed metadata features [ Silence ] and then I'll give you some best practices including some privacy [ Silence ] things to keep in mind and some other best practices. [ Silence ] So our first topic: metadata in AVFoundation..What kind [ Silence ] of classes are we going to be using to describe our metadata? [ Silence ] Well our primary model objects that we use [ Silence ] to describe both movie files or HLS streams is AVAsset. [ Silence ] AVAsset can contain any number of AV metadata objects [ Silence ] and each AVMetadataItem instance represents a single piece [ Silence ] of metadata, either your track name or your album mark, [ Silence ] even your location stuff that's going to be separate pieces [ Silence ] of metadata in our runtime environment. [ Silence ] So a closer look at AVMetadataItem: at its core, [ Silence ] it has two properties. [ Silence ] The first is identifier, which is actually a new property [ Silence ] and that is going to describe the kind [ Silence ] of metadata that you have. [ Silence ] In this example we have the song name and it's represented [ Silence ] by this long symbol name, [ Silence ] AVMetadataIdentifieriTunes MetadataSongName, [ Silence ] and then you have the value [ Silence ] which is the actual payload of the metadata item. [ Silence ] So for song name, it's the name of the song as a string. [ Silence ] As an example for cover art, [ Silence ] you can see that the value doesn't have [ Silence ] to be a string it can be an image or any other object [ Silence ] that supports both the NSObject and NSCopying protocols. [ Silence ] Now if you've used AVMetadataItem [ Silence ] in the past you might be familiar with the key [ Silence ] and key space properties. [ Silence ] Well the identifier I mentioned that was new, [ Silence ] it's new because it is a combination [ Silence ] of the old properties, key and key space. [ Silence ] So I'm not going to talking much about key and key space today [ Silence ] but mostly going to be talking about identifier going forward [ Silence ] as the way to describe your metadata. [ Silence ] Take a look at some of the built-in identifiers we have; [ Silence ] this is just a tiny sampling. [ Silence ] There's a lot of them. [ Silence ] You can find them [ Silence ] in AVMetadataIdentifiers.h. I've arranged them here according [ Silence ] roughly to the old notion of key space so that's just sort [ Silence ] of a sampling of the kind of metadata that we already know [ Silence ] that you might want to represent. [ Silence ] Going back to the metadata item itself, [ Silence ] we have a property that's also new called dataType, [ Silence ] which describes the native data type [ Silence ] that your metadata is representing. [ Silence ] So for the case of our song name it's stored as a string [ Silence ] so we see that the data type is a UTF8 string; [ Silence ] and these string constants [ Silence ] that represent the different data types are all defined [ Silence ] in CMMetadata.h. And besides the data type property, [ Silence ] we also have several type coercion properties [ Silence ] that you can use if you know you want to get your payload [ Silence ] in the form of a certain type of Objective C object. [ Silence ] So you have string value, number value, date value and data value [ Silence ] and those are going to give you exactly what you'd expect. [ Silence ] For the case of where our native payload is a string, [ Silence ] only string value is going to give you an interesting answer. [ Silence ] The rest will give you NULL. [ Silence ] For our artwork example where the payload is a JPEG image, [ Silence ] the top three are going to give you NULL [ Silence ] and the data value is going to give you the NSData [ Silence ] that you're looking for to grab the bytes of the JPEG image. [ Silence ] There are examples where you can have more [ Silence ] than one non-nil tech coercion method. [ Silence ] And one is creation date if you have the date represented [ Silence ] as a standard string format for dates. [ Silence ] You can either get the actual string that was stored that way [ Silence ] or you can ask the metadata item to give you an instance [ Silence ] of NSDate that describes the same thing [ Silence ] in a more convenient representation. [ Silence ] So that's your brief intro to AVMetadataItem, [ Silence ] we're going to be talking a lot about it throughout the talk. [ Silence ] Let's go back to AVAsset [ Silence ] so we can see how we actually get these metadata items. [ Silence ] So the easiest way is just to ask for all of the metadata [ Silence ] that applies to the entire asset. [ Silence ] There are types of metadata that apply to just parts of the asset [ Silence ] but this is how you get the metadata to, like, the location [ Silence ] and the song title that applies to the entire asset. [ Silence ] There's also a way to get just a subset of the metadata. [ Silence ] We have this notion of metadata format but I'm not going [ Silence ] to be talking about too much today but you can use it [ Silence ] to get just that subset of the metadata. [ Silence ] So for our example, when we are getting iTunes metadata, [ Silence ] we're going to use that AVMetadataFormatiTunesMetadata [ Silence ] and grab all of that using the metadataForFormat method. [ Silence ] And then from there we can use this filtering method, [ Silence ] metadataItemsFromArray, filtered by an identifier [ Silence ] to get just the items that correspond [ Silence ] to the song name identifier. [ Silence ] You might be wondering why you can have more [ Silence ] than one song name in a single asset? [ Silence ] We'll get back to that in just a little bit but first I want [ Silence ] to talk about how you load the payload of your metadata items. [ Silence ] AVMetadataItem conforms [ Silence ] to the AVAsynchronousKeyValueLoading.h [ Silence ] protocol. [ Silence ] This is a protocol we define ourselves in AVFoundation [ Silence ] and a lot of our core model objects conform to it [ Silence ] because a lot of times when you get an AVAsset [ Silence ] or an AVMetadataItem we haven't actually loaded the data behind [ Silence ] it yet. [ Silence ] So you can use this method, load values asynchronously for keys [ Silence ] to load the specific values you want and they'll do [ Silence ] that asynchronously so you're not blocking your main thread [ Silence ] with some sort of synchronous I/O or something like that. [ Silence ] So for this case we have our metadata item. [ Silence ] We're looking for the value so we just load the key value and, [ Silence ] when we get our completion handler, [ Silence ] we're going to check the status to make sure [ Silence ] that that loading succeeded and, assuming that we did, [ Silence ] we can then just go ahead and grab the value [ Silence ] and use it however we see fit. [ Silence ] So back to that whole multiple titles in one asset string. [ Silence ] Well, one reason we might have that is [ Silence ] if we have the asset localized in multiple languages. [ Silence ] So an asset can have the same metadata items [ Silence ] in multiple languages. [ Silence ] The example that we're going to talk [ Silence ] about is QuickTimeUserDataFullName [ Silence ] dat identifier. [ Silence ] If you use this identifier in your files, [ Silence ] then QuickTime Player, for example, can pick up the title [ Silence ] and display it in the title bar. [ Silence ] So this is just yet another example [ Silence ] of how metadata is used in our applications. [ Silence ] This particular example [ Silence ] of the movie actually has the title available in both English [ Silence ] and Spanish so here we have the English as the system language [ Silence ] so QuickTime Player picks that up, picks up the English title [ Silence ] but if we set our system language to Spanish it will pick [ Silence ] up the Spanish localization of that title instead. [ Silence ] These are represented as two distinct pieces of metadata [ Silence ] within the file and the way that you distinguish between them is [ Silence ] that they'll have different values [ Silence ] for these final two properties of AVMetadataItem locale [ Silence ] and extendedLanguageTag. [ Silence ] ExtendedLanguageTag is new in this release. [ Silence ] It's a BCP 47 language tag and it's particularly useful [ Silence ] when you want to distinguish written languages. [ Silence ] So that's one reason why you might have more [ Silence ] than one metadata item with the same identifier. [ Silence ] So I mentioned before that not all metadata applies [ Silence ] to the entire asset, well one example of that is metadata [ Silence ] that only applies to a particular track, [ Silence ] so for this example we have a special label attached [ Silence ] to our subtitle track called SDH, that stands for Subtitles [ Silence ] for the Deaf or Hard of Hearing [ Silence ] and that's basically just a more rich form of subtitles [ Silence ] that includes things like labeling who's talking [ Silence ] and mentioning sound effects [ Silence ] that are vital to the understanding. [ Silence ] We talked a little bit more about SDH and accessibility [ Silence ] in general last year in our "Preparing and Presenting Media [ Silence ] for Accessibility" talk so check that one out for more details. [ Silence ] For the purposes of this talk, [ Silence ] just know that to get this SDH label here, [ Silence ] it involves setting track-specific metadata. [ Silence ] So let's talk about how you actually find [ Silence ] out if your track has this metadata in it, [ Silence ] well you're going to use AVAsset track [ Silence ] and it has pretty much the exact same API [ Silence ] as AVAsset for reading metadata. [ Silence ] You have your metadata property; [ Silence ] you have your metadataForFormat method and so if we want [ Silence ] to find all the tagged characteristics that are [ Silence ] in an asset track, we're going to ask the track [ Silence ] for its metadata for the FormatQuickTimeUserData. [ Silence ] Once we have that we use [ Silence ] that same filtering method we saw before in order to get all [ Silence ] of the items that have the identifier, [ Silence ] QuickTimeUserDataTagged Characteristic. [ Silence ] So this is one example of tagged characteristics is the SDH [ Silence ] that I just talked about and it's the payload [ Silence ] of the metadata items that tells you what kind [ Silence ] of tagged characteristic you're dealing with. [ Silence ] We'll talk a little bit more detail about SDH [ Silence ] and how you author it in just a little bit. [ Silence ] So going back to our list [ Silence ] of identifiers you might have noticed some patterns [ Silence ] if you were looking closely. [ Silence ] Each of these groups has their own version of a title [ Silence ] or a song name or something like that. [ Silence ] We noticed that and come up with our own special kind [ Silence ] of identifier called a CommonIdentifier [ Silence ] which can be used when you want to look up, say, [ Silence ] for this example a title without caring exactly how it's stored [ Silence ] in your file. [ Silence ] Same for copyright here; we also have a common identifier [ Silence ] that represents copyright. [ Silence ] These are not the only common identifiers; [ Silence ] there's a whole list of them but these are just two examples. [ Silence ] So if we go back to our example where we're looking [ Silence ] for our iTunes song name, if we don't actually care [ Silence ] that the title of our asset is stored as iTunes metadata [ Silence ] and we just want a title so we can display it somewhere, [ Silence ] you can ask the asset for its array of commonMetadata [ Silence ] and this is all the metadata items [ Silence ] that can be represented using a common identifier. [ Silence ] Then you use that same filtering method we've been using [ Silence ] to filter down to just the ones [ Silence ] that have the CommonIdentifierTitle [ Silence ] and you can go from there with your title. [ Silence ] Also worth noting is [ Silence ] that AVAssetTrack has the same property, commonMetadata, [ Silence ] so you can do the same thing over there as well. [ Silence ] So that is your brief introduction [ Silence ] to inspecting metadata with AVFoundation. [ Silence ] Let's talk a little bit about authoring. [ Silence ] If you want to make your own files that have say location [ Silence ] or iTunes metadata in them, we have several different classes [ Silence ] that can write movie files, AVAssetExportSession, [ Silence ] AVAssetWriter and the capture movie and audio files outputs [ Silence ] and these all have the exact same redirect property called [ Silence ] simply, metadata. [ Silence ] So you give an array of metadata items and then [ Silence ] that will be written out to the file. [ Silence ] Similarly for track-specific metadata [ Silence ] like those tagged characteristics, [ Silence ] you can use an AVAssetWriter input [ Silence ] which also has the exact same property. [ Silence ] Now you are not limited to just writing out metadata [ Silence ] that you got from somewhere else, like another file [ Silence ] through the APIs we've been looking at. [ Silence ] You can also create your own metadata items [ Silence ] with a mutable subclass of metadataItem. [ Silence ] And as you might expect, this just has read/write properties [ Silence ] for all of the properties in AVMetadataItem. [ Silence ] So if we use an example of writing a subtitle track [ Silence ] that is marked as SDH, well it's actually two different tag [ Silence ] characteristics that you have to use [ Silence ] and so we'll create two different metadata items, [ Silence ] set both of their identifiers to the identifier we just saw, [ Silence ] the QuickTimeUserData tag characteristic, [ Silence ] but one of them will set the value [ Silence ] to TranscribesSpokenDialogue ForAccessibility [ Silence ] and the other will be DescribesMusicAndSound [ Silence ] ForAccessibility. [ Silence ] Then we get the subtitle AssetWriterInputthat's going [ Silence ] to write our subtitle track and set that array of the two items [ Silence ] on our asset writer input. [ Silence ] So that's how you would author a subtitle track [ Silence ] that is marked as SDH. [ Silence ] Just one example of using tag characteristics [ Silence ] in AVMutableMetadataItem. [ Silence ] Special note about AVAssetExportSession: [ Silence ] by default the ExportSession is actually going to take any [ Silence ] of the metadata that's in the source asset [ Silence ] that you're exporting. [ Silence ] It's going to copy that over to the output file. [ Silence ] Now that's not the case if you set metadata [ Silence ] on its metadata property. [ Silence ] That will be the signal to tell the ExportSession [ Silence ] to ignore the metadata in the source file [ Silence ] and instead write just what you put on the property. [ Silence ] So if you want to do augmentation of the metadata [ Silence ] or some other sort of modification you'll want to grab [ Silence ] that array of metadata, make a mutable copy [ Silence ] and do any adjustments that you want and then set [ Silence ] that on the metadata property. [ Silence ] So that's just a quick note about ExportSession. [ Silence ] The last note about authoring metadata is HTTP Live Streaming, [ Silence ] this is actually a new feature in iOS 8, OS X Yosemite [ Silence ] and you can use a new tag called session-data tag [ Silence ] in your playlist, which has two required fields: [ Silence ] the data ID which is a lot like our identifiers [ Silence ] that we're talking about, a URI which can point to the payload [ Silence ] or a value which directly specifies the payload and, [ Silence ] optionally, some language information. [ Silence ] So here's an example that shows very similar [ Silence ] to what we saw before with the titles [ Silence ] in two different languages but this is the markup you'd use [ Silence ] for HTTP Live Streaming. [ Silence ] So for more information on reading [ Silence ] and writing metadata we do have some sample code, [ Silence ] it's called AVmetadataeditor and for more information [ Silence ] about the details of writing HTTP Live Streaming metadata see [ Silence ] the documents at this URL. [ Silence ] All right so that is your crash course in metadata [ Silence ] in AVFoundation, our next topic is timed metadata. [ Silence ] So timed metadata, although I mentioned we have new features, [ Silence ] it is not a new concept. [ Silence ] We supported the notion of chapters for quite some time [ Silence ] and conceptually chapters are just an example [ Silence ] of times metadata. [ Silence ] Each of these chapter markers is just a piece of metadata [ Silence ] that is describing a particular range [ Silence ] of the timeline of the movie. [ Silence ] That's all that timed metadata is, [ Silence ] it's just metadata associated with a range of time. [ Silence ] So similarly, with our dynamic location example, [ Silence ] we have the path that's drawn here that's really just composed [ Silence ] of a number of pieces [ Silence ] of metadata indicating the current location, [ Silence ] each one of them associated with a particular time [ Silence ] in the movie's timeline. [ Silence ] So to demonstrate QuickTime Player's features [ Silence ] with dynamic location in Yosemite, [ Silence ] I want to bring my colleague, Shalini, [ Silence ] up to the stage for a demo. [ Silence ] >> Hi, I'm here to demonstrate how to read [ Silence ] and play back metadata using QuickTime Player. [ Silence ] Here I have a movie file which has both audio and video [ Silence ] and timed locations data stored in a different track. [ Silence ] So now if I bring this up in QuickTime Player, [ Silence ] this is the usual UI for audio and video. [ Silence ] New in OS X Yosemite: [ Silence ] in the Movie Inspector you can see a map view [ Silence ] if your movie file has location data. [ Silence ] Your map view is presented along with the route [ Silence ] where you have recorded this video. [ Silence ] So here the blue line indicates the path [ Silence ] where we recorded the video and the red pin is an indication [ Silence ] of the current location or the location [ Silence ] on the timeline of the movie. [ Silence ] So if I zoom in a little bit and start play, [ Silence ] you can see as the movie progresses the pin's location is [ Silence ] being updated to be in sync with the video. [ Silence ] I can drag the scrubber around [ Silence ] and you can see the pin moving back and forth. [ Silence ] I can also go and click at any point in the map [ Silence ] and you see the video seek to that location to present [ Silence ] where your video was when you were at that location. [ Silence ] This is map view in QuickTime Player on OS X Yosemite. [ Silence ] >> Thank you, Shalini. [ Silence ] So let's talk about what we just saw there. [ Silence ] So that location information was stored as timed metadata [ Silence ] in the file and in order to have QuickTime Player draw [ Silence ] that information on the map, we use AVAssetReader to read all [ Silence ] of the location information from that asset. [ Silence ] And because timed metadata is stored in its own track, [ Silence ] we use an AVAssetReaderTrackOutput to read [ Silence ] that data and we use a new class called AVAssetReaderOutput [ Silence ] MetadataAdaptor that knows how to give us that data in the [ Silence ] from of a class called AVTimedMetadataGroup. [ Silence ] Then from there we can grab each location [ Silence ] and draw that path on the map. [ Silence ] So AVTimedMetadataGroup is a very simple class. [ Silence ] It's really just these two properties: [ Silence ] an array of metadata items combined with a time range [ Silence ] that describes where in the movie that data applies. [ Silence ] So to see a little bit of code for using AssetReader [ Silence ] for this purpose, the first thing you want [ Silence ] to do is find the track [ Silence ] that contains your location information and we'll talk more [ Silence ] about how to do that in just a second. [ Silence ] Then you use that track to create an AssetReaderTrackOutput [ Silence ] and you use nil output settings [ Silence ] and then you'll create your metadataAdaptor [ Silence ] with that trackOutput. [ Silence ] And then, in a loop we just take your metadataAdaptor [ Silence ] and call the nextTimedMetadataGroup method [ Silence ] over and over again, doing something with each piece [ Silence ] of data, like drawing it on the map [ Silence ] until that method returns nil. [ Silence ] Then you know there's no more data to draw. [ Silence ] So in terms of finding the right track to read, [ Silence ] the way you're going to do that is by examining the tracks [ Silence ] in your asset and looking through the format description [ Silence ] of each track to find the identifiers you're looking for. [ Silence ] So you first start by getting the tracks [ Silence ] with the MediaTypeMetadata and then for each [ Silence ] of those tracks you're going to loop through all [ Silence ] of its format descriptions, usually there's only one [ Silence ] and for each format description you're going to grab its list [ Silence ] of identifiers using this function and check whether [ Silence ] that identifier array contains the identifier you're [ Silence ] looking for. [ Silence ] In this case we're looking [ Silence ] for the location ISO 6709 identifier. [ Silence ] So once we've found it we're good to go and we can resume [ Silence ] with the code on the previous slide. [ Silence ] So that's how QuickTime Player is drawing the map [ Silence ] or drawing the path on the map before you start playback. [ Silence ] The other thing that QuickTime Player does, as you saw, [ Silence ] is it can update the current location while you're doing [ Silence ] playback or even scrubbing around and the way it does [ Silence ] that while it's already playing the asset using an AVPlayerItem [ Silence ] and we're going to use a new class called [ Silence ] AVPlayerItemMetadataOutput that you attach to your PlayerItem, [ Silence ] which also notes how to vend this data in the form [ Silence ] of TimedMetadataGroups. [ Silence ] But unlike the asset reader, instead of getting all the data [ Silence ] up front you're going to be getting it piece [ Silence ] by piece as the movie plays. [ Silence ] So a little bit of code, [ Silence ] you first create your metadata output using the [ Silence ] initWithIdentifiers method [ Silence ] and in this case we're only interested in metadata that has [ Silence ] that location identifier so that's all we're going to get [ Silence ] by opting into this way. [ Silence ] Then you create a delegate that you define [ Silence ] and that's what's going [ Silence ] to receive the metadata during playback and you set [ Silence ] that delegate on your output [ Silence ] and tell us what cue you want us to send the data on. [ Silence ] Then you create or grab your AVPlayerItem and call addOutput [ Silence ] to attach your output, to attach your output to the playerItem [ Silence ] and finally make your player and associate your item [ Silence ] with the player as the current item and start playback. [ Silence ] It's important to get the smoothest playback experience [ Silence ] possible, we highly recommend that you do all of this sort [ Silence ] of setup work before you start playback [ Silence ] or even attach the item to the player. [ Silence ] So a little bit of look at what your delegate method might [ Silence ] look like. [ Silence ] There's only one delegate method; it's the metadataOutput, [ Silence ] didOutputTimedMetadataGroups, fromPlayerItemTrack method. [ Silence ] And the first thing you want to do is grab an item [ Silence ] that you can get your payload data from. [ Silence ] In this case, to keep things simple, [ Silence ] I'm just grabbing the first item from the first group but keep [ Silence ] in mind there could be multiple items, [ Silence ] there could even be multiple groups. [ Silence ] One reason there could be multiple groups given [ Silence ] to this method is that the metadata output will keep track [ Silence ] of whether the metadata is coming faster [ Silence ] than you're processing it and, if it is, it will start to batch [ Silence ] that up and give you the metadata in batches [ Silence ] when you're done with the previous batch of metadata. [ Silence ] So moving on with your item; you're going [ Silence ] to do this LoadValueAsynchronouslyForKeys [ Silence ] dance that we talked about before. [ Silence ] In this case, we're interested in the value [ Silence ] and data type properties so we're going [ Silence ] to load both of those. [ Silence ] I've admitted the error checking for brevity here [ Silence ] which you'll probably want to do that error checking [ Silence ] like we had in the other slide. [ Silence ] And once we have the completion handler we can ask the item [ Silence ] for its data type and make sure that's the data type we're [ Silence ] prepared to handle, in this case my code only knows how [ Silence ] to handle location information in ISO 6709 format so we got [ Silence ] to make sure that's the right data type and from there we go [ Silence ] and dispatch our code to the main thread [ Silence ] that will update our UI. [ Silence ] So that's how QuickTime Player is updating the location [ Silence ] metadata during playback. [ Silence ] Of course this is not the first API that we have offered [ Silence ] for reading timed metadata during playback. [ Silence ] There is an existing property called timedMetadata [ Silence ] on AVPlayerItem but I'm here to say [ Silence ] that the AVPlayerItemMetadataOutput [ Silence ] replaces that property for all of these use cases. [ Silence ] Now we're not deprecating the property yet [ Silence ] but we do recommend, if you're new to timed metadata, [ Silence ] just adopt the metadataOutput [ Silence ] and not worry about the property. [ Silence ] If you're already using the property version we do recommend [ Silence ] that you move over but just you know that you should make sure [ Silence ] that your code is working properly after that transition, [ Silence ] in particular I'll point [ Silence ] out that the metadataOutput will give you, for certain kinds [ Silence ] of HLS content, will give more specific identifiers [ Silence ] than the old property did. [ Silence ] So just make sure your code is prepared to handle that. [ Silence ] The last topic on reading timed metadata is Chapters. [ Silence ] Chapters, like I said, have been supported for some time; [ Silence ] they even have their own API: [ Silence ] chapterMetadataGroupsBest MatchingPreferredLanguages. [ Silence ] This is on AVAsset. [ Silence ] This will give you an array of timed metadata groups [ Silence ] that contain items with the identifier, [ Silence ] QuickTimeUserDataChapter, and we've supported this [ Silence ] for some time for QuickTime movie files and M4Vs and, [ Silence ] new in iOS 8 is the-and OS X Yosemite-is support for chapters [ Silence ] in HTTP Live Streams as well as MP3 files. [ Silence ] And I'll tell you more about how [ Silence ] to author those HLS chapters in just a little bit. [ Silence ] So for more information, we have some sample code [ Silence ] that does approximately what QuickTime Player is doing, [ Silence ] where it can show your location during play back. [ Silence ] We also have a previous session about AssetReader that goes [ Silence ] into much more detail than I did here, called "Working with Media [ Silence ] in AVFoundation" from 2011. [ Silence ] So that's how you read and play back timed metadata. [ Silence ] Our next timed metadata topic is how you can create your own [ Silence ] movies that contain timed metadata. [ Silence ] We saw the screenshot before and I mentioned [ Silence ] that these annotations are stored as timed metadata and, [ Silence ] to show you this demo app, I'd like to invite Shalini back [ Silence ] up on stage to demo it. [ Silence ] >> This time let's look at an app on how [ Silence ] to author your own custom metadata movie files. [ Silence ] Here I have a video and if I would like to share some notes [ Silence ] with my friend, who is good at fixing colors in a movie, [ Silence ] I can now do that within the app. [ Silence ] To add annotations, I use a two-finger gesture, [ Silence ] I can use a pinch gesture to resize and then add a comment [ Silence ] which is enough for my whoever looks at the video later [ Silence ] to fix the colors there and then I begin playback. [ Silence ] And as playback progresses, I track the circle [ Silence ] to where I want this to be fixed. [ Silence ] And now that I have this annotation and I can write it [ Silence ] out along with the audio and video to do that, I hit "export" [ Silence ] and now we see an AV player view controller [ Silence ] which shows the exported movie along with the metadata [ Silence ] which was written to it. [ Silence ] So if I start playback you see the annotation is moving along [ Silence ] the timeline in the part in which I traced. [ Silence ] So if I scrub back in time you can see the annotation moving. [ Silence ] You might wonder that the annotation is baked [ Silence ] into the video frame; it is not. [ Silence ] It is being rendered real-time using AVPlayerItemMetadataOutput [ Silence ] and you can change the color or the font of the annotation. [ Silence ] So if I begin playback, you see the rendering is happening [ Silence ] in real time. [ Silence ] That's AVTimedAnnotationWriter, we have this available [ Silence ] as a sample code as well, thank you. [ Silence ] >> So that was a great demonstration [ Silence ] of not only the playback part of it but also how to write [ Silence ] that data into the file, so let's take a look at how [ Silence ] that was accomplished. [ Silence ] So we're going to use an AVAssetWriter to write the file [ Silence ] and we're going to use an AVAssetWriterInput in order [ Silence ] to write that metadata track to the file. [ Silence ] Just like the reader side, [ Silence ] the writer has a new class that's a metadataAdaptor [ Silence ] and that class knows how to interpret instances [ Silence ] of AVTimedMetadataGroup and write that into the file. [ Silence ] See a little bit of code; first thing we're going [ Silence ] to do is create our AssetWriter Input. [ Silence ] We're going to use the media type AVMediaTypeMetadata, [ Silence ] once again nil outputSettings and we're going to have [ Silence ] to provide a clue to the source format, well, [ Silence ] the format of the data that we're going to be appending. [ Silence ] We'll talk more about this [ Silence ] and why it's required on the next slide. [ Silence ] Then you simply create your metadataAdaptor [ Silence ] with the reference to that input and, as you generate [ Silence ] or receive your timed metadata groups, [ Silence ] you simply use the appendTimedMetadataGroup method [ Silence ] to continue to append those and write them to the file. [ Silence ] So what's the deal with that source format thing. [ Silence ] Well, it turns out in order for AVAssetWriter to be able [ Silence ] to write your metadata in the most efficient way possible, [ Silence ] it needs to know up front exactly what kind [ Silence ] of metadata it is going to be writing. [ Silence ] This will result in the most lowest storage overhead in terms [ Silence ] of the number of bytes your file takes up [ Silence ] and it also has a effect on how efficient it is [ Silence ] to play back this kind of contents. [ Silence ] You don't want to be using too much power [ Silence ] when you're playing this kind of content back. [ Silence ] So you do have some options in terms [ Silence ] of how you actually construct one of these format hits. [ Silence ] If you're reading [ Silence ] from AVAssetReader you can actually ask the track [ Silence ] that you are reading from to give you its list [ Silence ] of format descriptions and use one of those. [ Silence ] If you're creating the metadata group yourself or getting it [ Silence ] from some other source then you can use a new method called [ Silence ] copyFormatDescription that will give you back an instance [ Silence ] of CM format description that will do this job for you. [ Silence ] It's important to note that if you go this route you need [ Silence ] to make sure that the contents [ Silence ] of your metadata group are comprehensive in terms [ Silence ] of it containing every combination [ Silence ] of identifier data type and language tag [ Silence ] that you are going to be appending. [ Silence ] That is, it contains an item with each of those combinations. [ Silence ] Of course, since the CM format description is a CF type, [ Silence ] you'll need a CFRelease app when you're done. [ Silence ] Of course, there's one more way you can do this: [ Silence ] you can create the format description directly using [ Silence ] CoreMedia APIs. [ Silence ] And here you use this long name CMMetadataFormatDescription [ Silence ] CreateWith MetadataSpecifications function. [ Silence ] You're going to pass in the metadataType box. [ Silence ] That's the sort of metadata we've been talking [ Silence ] about this whole time with timed metadata. [ Silence ] And these metadata specifications it's just an [ Silence ] array of dictionaries. [ Silence ] Each dictionary contains those combinations I was talking [ Silence ] about before. [ Silence ] The identifier dataType and optionally extended language tag [ Silence ] so you want to make one of these metadata specifications [ Silence ] dictionaries for each combination you plan to append. [ Silence ] So the one thing that was not obvious about that demo is [ Silence ] that we're actually writing metadata timed metadata [ Silence ] that describes one particular other track. [ Silence ] So for the example of these annotations, [ Silence ] we're really just talking about the video track of the movie [ Silence ] and not the sound or anything else like that. [ Silence ] So just like we had a way of making track-specific metadata [ Silence ] that applied to the entire track, [ Silence ] with those tagged characteristics [ Silence ] that we saw before, you also have the ability [ Silence ] to formerly mark your metadata track [ Silence ] as describing one particular other track. [ Silence ] You do that with the addTrackAssociationWith [ Silence ] TrackOfInput method using as the parameter the AssetWriterInput [ Silence ] that you are using to write your video track. [ Silence ] And your receiver is the input that you are using [ Silence ] to write your metadata track. [ Silence ] You use the AssociationTypeMetadataReferent. [ Silence ] So that's how your create metadata that's timed [ Silence ] but also specific to a particular track. [ Silence ] The next thing we did that was interesting in that demo is [ Silence ] that we actually used our own custom identifiers. [ Silence ] So we had that big list of built in identifiers. [ Silence ] Well, you don't have to use those; [ Silence ] you can actually build your own [ Silence ] and as I mentioned before an identifier is just a combination [ Silence ] of key space and key and it has a particular format: [ Silence ] it's just a string but it is in a particular format [ Silence ] so to help you make your own custom identifiers, [ Silence ] we have this method, identifierFor Key, and keySpace, [ Silence ] it's a class method on AVMetadataItem. [ Silence ] There are some rules to follow: your key space needs [ Silence ] to be four characters long if you want to use it [ Silence ] for timed metadata [ Silence ] so we actually recommend you use our built-in key space, [ Silence ] the QuickTimeMetadata keySpace. [ Silence ] We also highly recommend you use reverse DNS notation [ Silence ] for your custom keys to avoid collisions [ Silence ] with other kinds of metadata. [ Silence ] So a brief code snippet you can see you can simply use this [ Silence ] method to make your custom identifier and then set [ Silence ] that on the identifier property of your mutableMetadataItem. [ Silence ] So in addition to custom identifiers you can also create [ Silence ] your own custom data types. [ Silence ] So we're all familiar by now, through this presentation, [ Silence ] with some of the built-in data types that we defined; [ Silence ] there's a lot more than these [ Silence ] but we've been using these quite heavily already. [ Silence ] These are really useful [ Silence ] but sometimes you want your data type information [ Silence ] to express more, maybe about the domain you're working in, [ Silence ] so if you are doing a serial number or a bar code kind [ Silence ] of thing you might want [ Silence ] to define a data type that's this sort of serial number [ Silence ] as string data type or barcode image as JPEG data type [ Silence ] so you have more specific information [ Silence ] about what your metadata actually contains. [ Silence ] The way that this works is, you have to tell us exactly how [ Silence ] to serialize that custom data type and the way you do [ Silence ] that is you tell us that your custom data type conforms to one [ Silence ] of our built-in data types. [ Silence ] So in this case the serial number conforms [ Silence ] to the UTF8 data type so under the hood it's UTF8 string, [ Silence ] but we know that it really represents a serial number [ Silence ] and the same with the barcode image. [ Silence ] The way that you do this is you register your data type using [ Silence ] the CMMetadataDataTypeRegistry RegisterDataType function that's [ Silence ] defined in Core Media. [ Silence ] You can't create your own custom base types [ Silence ] but you can create your own custom type that conforms [ Silence ] to our raw data built-in type [ Silence ] if your data type really is just a custom sequence of bytes. [ Silence ] So there are some rules to using AVAssetWriter [ Silence ] for writing timed metadata. [ Silence ] Most importantly, every metadata item that you append has [ Silence ] to have non-nil values for identifier, data type and value. [ Silence ] Your identifier has to conform to the format that we specify, [ Silence ] so we highly recommend using that utility method [ Silence ] that we just talked about. [ Silence ] The value has to be compatible with the data type [ Silence ] so you can tell us that your NSString value is an UTF8 string [ Silence ] but don't try telling us [ Silence ] that your custom class is a UTF8 string because we won't know how [ Silence ] to serialize that properly and the AssetWriter will fail. [ Silence ] As I mentioned before, you have to create your AssetWriterInput [ Silence ] with a format hint and that must be comprehensive [ Silence ] and we described that before. [ Silence ] So the last topic about AssetWriter [ Silence ] and timed metadata is a recipe for creating your own movies [ Silence ] that have the same sort of dynamic location [ Silence ] that we've seen a couple of times already. [ Silence ] To do this, you can use AVCapture audio [ Silence ] and video data outputs and target that data [ Silence ] at twin instances of AssetWriterInput and, [ Silence ] at the same time, grab information from Core Location [ Silence ] that represents the location information and write [ Silence ] that to its own AssetWriterInput. [ Silence ] For more detail about how to do that we've actually implemented [ Silence ] that and made it available as sample code, [ Silence ] so see AVCaptureLocation if you want to make your own movies [ Silence ] that contain dynamic location. [ Silence ] We also have sample code as Shalini mentioned [ Silence ] for the demo we just showed you, [ Silence ] that's called AVTimedAnnotationWriter. [ Silence ] And, of course, for more information about AssetWriter [ Silence ] in general, see that same talk I referenced earlier: [ Silence ] "Working with Media in AVFoundation". [ Silence ] Last two quick topics about timed metadata: ExportSession. [ Silence ] Just like we've said the asset ExportSession will [ Silence ] by default pass through any of your metadata that applies [ Silence ] to the entire asset or entire track, it will pass [ Silence ] that through, copy it to the output file. [ Silence ] It will do the same thing with timed metadata that exists [ Silence ] in the source file provided [ Silence ] that your destination file type is QuickTime Movie. [ Silence ] We'll talk more about file types in just a little bit [ Silence ] but basically ExportSession behaves exactly [ Silence ] as you would expect. [ Silence ] In our last timed metadata authoring topic is HTTP Live [ Silence ] Streaming chapters so if you want to author chapters [ Silence ] in your HLS stream, you can use the session-data tag we talked [ Silence ] about earlier and the special data ID, com.apple.hls.chapters. [ Silence ] Your URL should point to a JSON file [ Silence ] that describes the chapter information for that stream and, [ Silence ] of course for more detail on this, see that same link [ Silence ] that I referenced earlier for HTTP Live Streaming. [ Silence ] All right, so that is timed metadata, [ Silence ] our next topic is privacy. [ Silence ] Why is privacy important in this context? [ Silence ] Well, any time that you are writing your users data [ Silence ] to a file you need to be at least considerate [ Silence ] about their privacy and be aware that the metadata that you write [ Silence ] out to these movie files can contain user identifiable [ Silence ] information, the most obvious example of that is location. [ Silence ] And so because movie files can be distributed and we want [ Silence ] to protect the privacy of our users, [ Silence ] for our built-in sharing services, we do our best [ Silence ] to strip out any potentially user identifiable information, [ Silence ] such as this location and we recommend that you do the same. [ Silence ] So we've given you a utility [ Silence ] for that called AVMetadataItemFilter. [ Silence ] Right now there is only one filter that we make available [ Silence ] but it is geared towards privacy, [ Silence ] it is the metadata item filter for sharing and that will strip [ Silence ] out any of this sort of user identifying information [ Silence ] that we're talking about; location is only one example. [ Silence ] But it will also strip out anything it doesn't recognize, [ Silence ] because it doesn't know whether [ Silence ] that might contain user identifiable information. [ Silence ] So that includes any metadata that uses identifiers [ Silence ] that you define yourself. [ Silence ] It will leave in some things like metadata that's important [ Silence ] to the structure of the movie [ Silence ] and chapters are the best example of that, [ Silence ] and also any commercial related data like your Apple ID. [ Silence ] So to use the MetadataItemFilter you're going to first [ Silence ] of all create your filter and feed it your original array [ Silence ] of metadata items using this metadataItemsFromArray [ Silence ] filteredByMetadataItemFilter method. [ Silence ] This is a companion to that other filtering method based [ Silence ] on identifiers we've been using all day [ Silence ] and then once you have your filtered array [ Silence ] of metadata items just set that on your AssetWriter [ Silence ] or ExportSession as you normally would. [ Silence ] Well actually I mentioned ExportSession [ Silence ] but things can be simple if you're using the ExportSession [ Silence ] and only want to copy the metadata [ Silence ] from the source asset and not add your own. [ Silence ] You just set the filter on the ExportSession [ Silence ] and it will actually do the filtering for you, [ Silence ] this will filter both static and timed metadata [ Silence ] but it will only filter the metadata from the source asset. [ Silence ] If you set your own metadata on the metadata property, [ Silence ] it won't filter that for you; you'll need to do the process [ Silence ] that I just described of doing the filtering yourself. [ Silence ] The only other thing to keep in mind is [ Silence ] that the export may take more time [ Silence ] when the filter is being used because it has to go through [ Silence ] and examine all of the metadata items. [ Silence ] So that's privacy. [ Silence ] Our last section of the talk today is some assorted best [ Silence ] practices when you are writing your own files [ Silence ] that contain metadata. [ Silence ] First up, what if you're writing timed metadata [ Silence ] and you have multiple streams of metadata [ Silence ] that use different identifiers. [ Silence ] How do you get those into the same file? [ Silence ] Well, we actually have the situation in the demo app, [ Silence ] we have that circle is comprised of two different pieces [ Silence ] of information, the position and the radius. [ Silence ] So we're representing these [ Silence ] and the demo app is two distinct streams of metadata. [ Silence ] And so the most obvious way I can think of to get this [ Silence ] into a file is to use two different AVAssetWriterInputs, [ Silence ] which result in having two metadata tracks [ Silence ] in the output file, pretty simple. [ Silence ] But there is another way you can do it, [ Silence ] you could instead combine those two different types of metadata [ Silence ] into one timed metadata group and write [ Silence ] that to a single AssetWriterInput [ Silence ] and that will result in only one metadata track [ Silence ] in the output file that contains multiple different kinds [ Silence ] of identifiers. [ Silence ] There are some advantages to this approach, not the least [ Silence ] of which is it can result in lower storage overhead [ Silence ] and therefore as we always see more efficient playback. [ Silence ] But there are of course pros and cons to everything. [ Silence ] So you'll definitely want to consider combining [ Silence ] into one track your different metadata [ Silence ] if they are used together during playback [ Silence ] and they have identical timing. [ Silence ] This is definitely the case with the example we just saw [ Silence ] with the circle center and the circle radius. [ Silence ] If these are not true then you might not want to combine. [ Silence ] And in fact one instance where you definitely do not want [ Silence ] to combine, is if you have one type [ Silence ] of metadata that's associated with another track in the file, [ Silence ] so that's like our annotations are associated [ Silence ] with the video track, but then you have another type [ Silence ] of metadata like location that is associated [ Silence ] with the entire asset. [ Silence ] You don't want to combine those into one track, [ Silence ] otherwise your location in that example will become mistakenly [ Silence ] associated with just the video track, [ Silence ] and that's not what you want. [ Silence ] So that's how to deal [ Silence ] with multiple streams of timed metadata. [ Silence ] Next topic is duration of your timed metadata groups, [ Silence ] when you get a timed metadata group [ Silence ] from AVFoundation it's always going [ Silence ] to have a fully formed time range. [ Silence ] So that means it will have a start time and a duration. [ Silence ] We actually recommend when you make your own timed metadata [ Silence ] groups for a pending with the AVAssetWriter [ Silence ] that you don't bother giving us a duration. [ Silence ] And to see how that works, here's an example of a group [ Silence ] that starts at time 0 but it doesn't have a duration [ Silence ] so how do we know when it ends? [ Silence ] Well, of course we'll wait until you append the next one [ Silence ] and then we'll say that, "Okay, the end time [ Silence ] of the first group is the same [ Silence ] as the start time of the next one." [ Silence ] So this ensures that your metadata track is going [ Silence ] to have a continuous stream of contiguous metadata and we think [ Silence ] that for most cases this is the best way to store your metadata. [ Silence ] The way you accomplish this is, [ Silence ] when you're making your time range, [ Silence ] you just use KCMTimeInvalid for your duration [ Silence ] and we'll take care of the rest. [ Silence ] We do recognize that there are cases where you might not want [ Silence ] to have contiguous metadata, you might want [ Silence ] to author an explicit gap into your metadata stream and so, [ Silence ] for that, our recommendation is that you give us [ Silence ] in the middle there a group that contains zero items. [ Silence ] This is the best way to author a gap in the metadata. [ Silence ] And you can see we just do that by presenting an empty array [ Silence ] when we're creating our timed metadata group. [ Silence ] Notice that we're still using KCMTimeInvalid [ Silence ] for our duration here. [ Silence ] Just tell us when the beginning of the metadata silence, [ Silence ] so to speak, is and we'll figure out how long it lasts based [ Silence ] on when you append your next non-empty group. [ Silence ] So that's how you write gaps in your metadata. [ Silence ] Our last best practice, [ Silence ] I mentioned output file type before [ Silence ] and here's the longer explanation. [ Silence ] Well, AssetWriter [ Silence ] and AssetExportSessions support writing [ Silence ] to a wide variety of file types. [ Silence ] You've got QuickTime movie, MPEG4, and all sorts [ Silence ] of other kind of file types [ Silence ] and those file types can carry different kinds of metadata; [ Silence ] some have more restrictions than others about what kind [ Silence ] of metadata can go into that file type. [ Silence ] So the easiest situation, say if you have an ExportSession [ Silence ] and you're going from one, from the same file type [ Silence ] as your source to the output, [ Silence ] so for this example they're both QuickTime movie files. [ Silence ] This is the easiest way to ensure that all [ Silence ] of that data is actually going to make it into the output file. [ Silence ] If instead you're using a different output file type, [ Silence ] like MPEG4 in this example, [ Silence ] then some different things are going to have to happen. [ Silence ] You notice those last few items didn't quite make it [ Silence ] into the output file; [ Silence ] it's because they have no equivalent representation [ Silence ] that works with an MPEG4 file. [ Silence ] If you're looking closely you'll also notice [ Silence ] that those top two items have changed, [ Silence ] although they sound very similar they are slightly different [ Silence ] identifiers because that's the kind of identifier [ Silence ] that works with MPEG4. [ Silence ] So both AssetExportSession and AssetWriter will do the sort [ Silence ] of three step process. [ Silence ] First, they'll try to pass that data through directly [ Silence ] if possible and, if not, they'll try to convert the identifier [ Silence ] into an equivalent representation [ Silence ] in the output file type. [ Silence ] If neither of those work, we have no choice but to just drop [ Silence ] that piece of metadata on the floor. [ Silence ] So in terms of guidance on how to choose an output file type, [ Silence ] well, my two recommendations are, [ Silence ] if you are using say an ExportSession to copy all [ Silence ] of the metadata, timed or otherwise, from the source asset [ Silence ] to your destination file, the best way is to try [ Silence ] and use the same file type that you started with, [ Silence ] and if you don't know what the file type is you can use the [ Silence ] NSURLTypeIdentifierKey to find out. [ Silence ] You can also always use the QuickTime Movie file [ Silence ] because that is going to have the greatest chance [ Silence ] of supporting your metadata no matter where it came from. [ Silence ] If AVFoundation supports it, there's a good chance [ Silence ] that it will be supported by the QuickTime movie file. [ Silence ] Of course, this is the only way [ Silence ] if you're writing timed metadata, [ Silence ] to get your timed metadata into a file is [ Silence ] to use QuickTime Movie file; it's the only file form [ Silence ] that supports it right now. [ Silence ] Of course good advice is always [ Silence ] to check the results, no matter what. [ Silence ] Check that your output files contain the kind of metadata [ Silence ] that you expect, all the metadata that you expect [ Silence ] and you can choose to use some of the APIs [ Silence ] that we've already talked about if you want [ Silence ] to do that at runtime. [ Silence ] Some guidance if that doesn't end up being the case: [ Silence ] if you don't get all the metadata that you expect, well, [ Silence ] you can try to do the conversion yourself. [ Silence ] Especially if you have a custom identifier and are going [ Silence ] to a file type that doesn't support your custom identifier, [ Silence ] take a look at that long list of built-in identifiers we have [ Silence ] and see if there is something that's roughly equivalent [ Silence ] to what you're trying to store and you can do [ Silence ] that conversion yourself. [ Silence ] One particular example I want to call [ Silence ] out that involves only built-in identifiers is [ Silence ] when you're trying to go from ID3 to iTunes, [ Silence ] well AVFoundation currently isn't going to do [ Silence ] that conversion for you. [ Silence ] But there's no reason you couldn't do that yourself, [ Silence ] so once again just take a look at our long list of identifiers [ Silence ] and match them up and do the conversion in your own code. [ Silence ] So that is the end of the talk. [ Silence ] See what we covered: we talked obviously a lot about metadata [ Silence ] in AVFoundation, we talked about all [ Silence ] of the different classes you can use for inspection, [ Silence ] we talked about AVAsset and AVMetadataItem [ Silence ] and how those work together and also authoring, [ Silence ] we talked about the AssetWriter, [ Silence ] the AssetExportSession even briefly on the capture audio [ Silence ] and movie file outputs. [ Silence ] We dove into timed metadata, including all the new features [ Silence ] that enable things like the dynamic location [ Silence ] and your own timed metadata like the annotation demo. [ Silence ] We also talked about privacy considerations [ Silence ] and some best practices like how to choose the right file type. [ Silence ] So for more information, you can contact our evangelism team [ Silence ] or see our programing guide, [ Silence ] there are some other related sessions you might be [ Silence ] interested in. [ Silence ] If you missed this morning's presentation [ Silence ] on "Modern Media Playback", you can catch [ Silence ] that on the video recording. [ Silence ] Tomorrow there is also a camera capture talk focusing [ Silence ] on manual controls. [ Silence ] And on Thursday we'll have a talk about direct access [ Silence ] to video encoding and decoding, which I'm sure a lot [ Silence ] of you will be interested in. [ Silence ]