Transcript
[ Music ]
[ Applause ]
>> Morning.
Good morning.
My name's Eric Hanson.
I'm the Technology Evangelist
for the Photos platform.
And, I'm joined by three of my
colleagues from the Photos
engineering team today to talk
to you about integrating your
application with Photos on
macOS.
We have two key things that
we're going to talk about.
The first is a full update on
the Photos Project Extension API
that we introduced last year in
macOS High Sierra.
And then, we're going to get
into interacting with your
application via drag and drop.
But first, let's talk about
Photo Project Extensions.
When iPhoto was first launched
in 2002, Apple became one of the
first companies to allow people
to create beautiful books, and
later cards and calendars.
To date over 70 million photo
books, cards, and calendars have
been created using iPhoto and
Photos.
But, in that same amount of
time, 16 years, we've seen this
blossoming market.
There is a tremendous breadth
now of choice for users to
create all sorts of things with
their photos, both physical and
digital.
And, it's the observance of this
amazing ecosystem that led us to
the introduction of the Photo
Project Extension API last year,
to build on that ecosystem.
We've seen some great extensions
already.
For example, Mimeo Photos.
It lets you create these books,
and cards, and calendars of
exceptional quality.
There's Whitewall, that lets you
create gallery-quality framed
prints from your photos.
And, we have digital services
like Wix.com, that make it very
easy for you to create web photo
albums to share with your
friends and family.
And, we're now seeing some new
extensions on the horizon.
Like this one, called Motif.
It is a brand-new fully native
experience, integrated directly
into Photos, coming to you this
summer.
All of this choice, this whole
ecosystem is a tremendous
opportunity to you, as a
developer.
You can take the very rich
metadata and the context and the
image curation that we pass to
your extension, and you can
build on that, using the full
stack of native frameworks on
macOS, to create something
really extraordinary, and new,
that delights users around the
world.
And, it's for this reason that
Apple is announcing we will be
transitioning our entire print
product business to this
ecosystem in macOS Mojave.
Project Extensions will be the
way users will create books,
cards, calendars, all sorts of
things.
Whatever you surprise us with
with your extensions.
And, by doing this, we're
building a better Photos
experience for everyone.
So, let's talk about what's new
in the Extensions API.
First, with the UI.
We now take the UI of your
extension, and integrate it
directly into the Photos app.
So, the sidebar that you're
familiar with is always
available.
Right? A user can be creating a
project in one of your
extensions, and can grab content
from the sidebar, and just drag
and drop it right onto your
project.
Or, they could pop up to the
photo search, search for a photo
they want to use, copy it, and
paste it into the project that
they're working on.
We're also allowing you to
integrate directly with the
powerful editing tools of
Photos.
For example, you may want to
allow a user to simply
double-click on a photo to edit
it.
And, you can do that now, with a
single line of API.
You can invoke the actual full
editor of Photos.
We pushed the edit session to
the top of the view stack, with
that photo loaded.
Let the user make any
adjustments they want to make.
And, we also give them access to
all of the other assets that are
used in that project.
When they're done with their
editing session, they simply hit
the Done button, and they're
returned to the project that
they're working on.
You get a notification that the
library changed.
You can respond to that, and
update your UI accordingly.
But, none of this would matter
if it was hard to find the apps
and the extensions that you
create.
And so, we're stepping up the
game in the discoverability of
your applications.
The Photos app links to the Mac
App Store in the Create menu.
And now, with the brand-new Mac
App Store, on macOS Mojave, we
actually can link directly to a
fully curated story that's
targeted specifically at the
extension experience.
And, this is an evergreen story
that will live on.
It's something that we can
update, and feature new
experiences, and educate users
on new ways to do things with
their photos.
And, when a user downloads an
app from this story, they're
downloading an app that contains
an extension.
And, historically, that may have
led you, as a developer, to put
some time into, kind of,
educating users in your
standalone app on how to use the
extension in Photos.
But, we think we can do
something better.
Wouldn't it be great if the user
could download an app and
immediately launch into the
project creation experience in
Photos?
We're making that possible.
We've added a custom URL scheme
to Photos, where you can simply
pass in your extension
identifier, and optionally pass
in a category.
And, Photos will be brought to
the back-- brought to the
foreground, with the options of
your extension displayed for the
user.
So, they're immediately taken
into the creation experience as
a first-launch experience.
And, we think this is great.
And, finally, when they create a
project, we want those projects
to live where they belong, right
inside the user's Photo library.
And so, that's exactly what we
do.
We have this Project Gallery in
Photos that lets a user see all
the things that they've created.
And we now let the extension
create a custom preview for each
project.
So, that preview can represent
what the project really is.
Take this photo mug for example.
Not only can a user double-click
on one of those projects to get
back into your extension, to
continue to work on it, they can
create projects from other
projects.
They can simply select a
project, go to the Create menu,
create something new from it.
And, we pass over everything we
can to help your extension, kind
of, start with some great
context.
But, in the case of the Apple
projects, we can do a lot more.
And so, we do.
We send you photo-by-photo,
page-by-page exactly how that
was laid out.
So that your extension can
create a complete conversion
experience from the Apple
projects.
And, we highly encourage you to
do this.
So, that's just a very
high-level, from a UI
perspective, of some of what's
new.
But, we'd like to take you a
little deeper.
And, to do that, I'd like to
invite to the stage, my
colleague from Photos
engineering, Tobias Conradi.
Tobias.
[ Applause ]
>> Good morning.
Thank you, Eric.
Hello, my name is Tobias
Conradi.
I'm an engineer on the Photos
team, and I will go into detail
for some of the changes we did
around Photos Project
Extensions.
First up, the Create menu.
When we first introduced Photos
Project Extensions in macOS High
Sierra, we put all extensions as
a flat list into the Create
menu.
But, as it turns out, in some
cases, it's kind of hard to
guess, just from the name of the
extension what kind of projects
you can create with that.
We want to improve the user
experience, so we are
introducing project categories.
The new Create menu in macOS
Mojave looks something like
this.
We have categories, submenus in
the Create menu.
And, the categories we're
introducing this year are Books,
Calendars, Cards, Wall Decor,
Prints, Slideshows, and for any
extension that doesn't fit into
these categories, Other.
Now the user can go into the
menu with a very specific
intent.
For example, I want to create
wall decorations, and these
[inaudible] all extensions that
support projects in that
category.
How does the extension show up
in the category?
We have a new key in the
extensions attribute dictionary
in the extensions info.plist.
It's called PHProjectCategory.
And, the value for the key, is a
list of the supported project
categories.
In this case, it's Wall Decor
and Other.
The next thing the user sees
from the extension, is the
[inaudible] sheet.
You provide us with the data,
and we display the project type
descriptions you provide.
The problem with the API we
introduced last year is that
it's kind of hard to provide
up-to-date price information or
current offers.
That's why we're introducing a
new, improved API to allow for
dynamic updates, which would
look like this.
In addition to dynamic updates,
we now also support display of a
custom footer text at the bottom
of the sheet, which you can use
to display legal text if you
have the requirement to do so.
On an API level, the new dynamic
API looks like this.
We have the new method on the
projectExtensionsController
protocol, and we ask for a data
source, instead of a list of
product type descriptions.
And, pass in the category from
the menu in which the
selection-- the extension was
selected.
And also, an invalidator object.
Once you return the data source,
we ask the data source for the
project type descriptions, and
for the optional footer text.
Whenever your extension has the
need to invalidate the
information returned, you can
use invalidator to invalidate
the project type descriptions,
or the footer text, and when
necessary, photos will refetch
the data from the data source,
and display the up-to-date data
in the UI.
The next topic is the
projectInfo.
The projectInfo is structured,
additional information about the
contents of the project.
And, it's structured into
sections, and sectionContents
which reflect creation level.
And, the elements have basic
layout hints, and also weights
and scores for the assets.
And, assets can contain
important regions in the asset's
regions of interest.
This is just a quick reminder
what the projectInfo looks like.
If you want to know more about
the projectInfo, I highly
recommend last year's session,
What's New in Photos API.
The projectInfo is handed to
your extension during begin
project.
And, whenever the user adds new
assets to the project, the
projectInfo will be outdated,
because it's a static object.
And, we want to solve the
problem by introducing new API
on the projectExtensionsContext
to get updated project info,
with the current state of the
project.
Let's take this as an example.
We handed you this projectInfo
during project creation, and
then the user added some more
assets to the project.
You can then call the
updatedProjectInfo method, with
this projectInfo, and we will
update any existing sections,
and append a new section for any
added assets.
If the user adds some more
assets, you can do the same
again.
Pass in this project info, we
will update the existing
sections, and append a new
section for any added assets.
Let's now go into a more
detailed element of the
projectInfo, the regions of
interest.
The regions of interest
highlight important regions in
the asset.
For example, the faces of
people.
And, if the same person appears
in multiple images, the region
of interest identifier will be
the same for the same person.
Let's focus on Person B for a
moment.
In addition to the region of
interest identifier, we have
weight on the regions of
interest, which represents the
importance of the region of
interest in the project.
But, if you want to decide which
image to pick as a
representative image for a
person, or for a region of
interest, it's kind of hard to
do with this API.
That's why we're adding a new
quality score, which represents
the quality of the region of
interest in the asset.
On the left-hand side, the
Person B is kind of out of focus
and not really central to the
image, while on the right-hand
side, he's really in focus.
That's why the quality score on
the right-hand side is higher
than on the left-hand side.
On an API level, that looks like
this.
We have this alt weight, which
is the importance of the region
of interest in the context of
the project.
And then, the quality in the
asset.
I would now like to show you a
quick demo, how you can use
regions of interest in your
extension to improve the
experience, and also listen to a
library notification to get
notified whenever the assets or
the project changes.
And, use a new updateProjectInfo
API.
OK. I created a slideshow
extension, and to show you how
it looks like, I selected some
assets in an album, and will
create a new project with the
extension.
The extension has two views, one
is the overview, which contains
all assets of the project.
And, the other is the playback
modus, which plays the
slideshow.
As you might notice, the
slideshow always zooms into the
center of the image, which is
kind of boring.
And, I want to use regions of
interest to always zoom into
interesting regions in the
photo.
So, let's switch to Xcode to fix
that.
Here in my asset model, I have
this property,
preferredZoomRect, and it always
returns the same rect, which is
kind of boring.
So, let's replace it with the
code.
What I want to do is to get all
regions of interest from the
asset element, and then sort
them by their weight and
quality.
And, from the sorted list of
regions of interest, we want to
get the last element, and return
its rect.
If we now rerun the extension--
We always zoom into the most
important region of interest in
the asset.
I think that looks way better.
[ Applause ]
So, next, I want to add some
more assets to the project.
I grab an album in the sidebar
and drag it over the extension.
There's a plus next to the
cursor, so it looks like the
extension accepts the drag.
I release the mouse and nothing
happens.
The assets were added to the
project, but my extension
doesn't list the change
notification.
And, doesn't know that assets
were added to the project.
So, let's switch back to Xcode
and fix that.
In my projectViewController, I
have the begin and resumeProject
method, which are a part of the
projectExtensionController
protocol.
And, here I want to register for
change observation.
And, we add the same code in
both methods.
First, we get the PhotoKit
object we're interested in.
We're interested in additions of
assets to the project, so we are
getting a fetch result for all
assets on the project, and then
we are registering asset change
observer for the library.
In finished project, we
[inaudible] registering.
And, since Xcode complains that
we don't implement the protocol,
we add that.
Change observer.
And, down here--
We implement
photoLibraryDidChange.
PhotoLibraryDidChange is called
whenever anything changes in the
photo library, and we get a
change instance as a method
argument.
We can ask the change engine
instance for changeDetails of an
object we're interested in.
We're interested in the changes
of the fetchAllAssets on the
project, so we pass that
fetchResult in.
And, if change results are
returned, we update our locally
cached fetch result with the
fetchResultsAfterChanges, and
get the projectExtensionContext,
and call updatedProjectInfo.
And pass in the project info we
have.
Once the updatedProjectInfo is
returned to us, we call the same
setup code, but you could do
something more sophisticated in
here.
Let's rerun the project again.
OK. I can now drag the album
from the sidebar, release it
over the extension, and the
assets are added to the project,
and the extension listens to the
photo library change
observation, gets notified of
the change, and we update our
project info.
[ Applause ]
So, as you just saw, with only a
few steps, I was able to improve
my extension by using the
projectInfo, and the regions of
interest in the projectInfo, and
also registering asset change
observer to get notified
whenever anything changes in the
photo lobby, and also update my
project info.
There are two things I want to
highlight with integration.
By default, Photos handles copy
and paste of assets or albums
from Photos into the extension.
But, if your extension also
wants to implement the paste
section, which you should
probably do for text and things
like that, we need your help to
know when your extension should
handle the paste section, and
when Photos should try to handle
the paste section.
So, please implement
validateMenuItem, and if the
menu action is a paste action,
check if your extension can
handle the current pasteboard
contents, and if no, return
false, and we will try to handle
it as Photos.
And, otherwise, if you can
handle it, return true, and your
photos will get the paste
action.
Something similar applies to
drag and drop.
Photos handles any drags of
PhotoKit objects to the
extension by default, but your
extension can potentially
interfere with that if you
register for the wrong types.
So, please be careful what types
your extension is registering
for, and only register for
extension internal drags, or
other drags you really want to
handle.
Be especially careful if you use
WKWebView, because it registers
for a bunch of drag types by
default.
That was everything about photo
project extensions.
Let me now hand over to my
colleague Sanaa to talk about
integrating with Photos--
interacting with Photos as a
third-party application.
Thank you.
[ Applause ]
>> Thank you, Tobias.
Hello, everyone.
My name is Sanaa.
I am a Photos engineer, and
today with my colleague Joachim,
we are going to talk about some
best practices for receiving and
providing images or videos
between apps on macOS, using
drag and drop.
Drag and drop is one of the most
intuitive and easiest way to
move items from one place to
another.
But, sometimes, you might have
encountered this situation
instead.
If this is something happening
with your app, then you are
sitting in the right session.
So--
[ Applause ]
Let's take a step back to
understand what's happening
here.
Drag and drop on macOS is using
NSPasteboard, and when it's
wrapped in with the pasteboard,
reading and writing data happens
on the main trade, for both the
receiving and providing apps.
In the past, since all local was
stored on local disk, you could
put the file URL's into the
pasteboard.
But, things have changed today.
Images or videos might not be on
disk if the user is using
iCloud, so Photos, we need to
download the full resolution
item first, before putting the
file URL into the pasteboard.
Also, Photos respect the privacy
settings of drag and drop, so if
the user choose to save the
location information, then
Photos will export a new file
which does not contain this
metadata.
Both downloading and exporting
file takes time, and you don't
want to do that on the main
trade, since it's going to block
your app UI.
So, in order to fix this, we
need an asynchronous API.
In fact, we do have one, it's
called file promises.
File promise is a promise of--
that's a file of a certain type
that does not exist yet on disk,
will be written in a provided
location.
It also allows the sender to
write files in the background.
There are two ways to interact
with file promises: receiving
files by using
NSFilePromiseReceiver and
providing files by using
NSFilePromiseProvider.
Both of these modern API's have
been introduced two years ago
with macOS Sierra.
So, let's start first with
receiving file promises.
As a general rule, apps
supporting drag and drop should
always accept both file URL's
and file promises.
And, I'll explain why.
There are multiple apps
providing file promises.
Photos is using file promises
when dragging an image, a video,
or entire album, and since macOS
Mojave, you can-- we added the
ability to drag the people or
memories.
So, we are not the only app
using file promises.
Mail is using file promises when
dragging a message to Finder, to
save the entire email, including
attachments.
Safari is using file promises
when dragging an image.
And, Keynote is using file
promises when dragging a
selection of slides to create a
new document containing those
slides.
So, if you want to receive files
from those apps, or any app
providing file promises, then
your app needs to read file
promises, and need to accept
those files.
So, let's see how we can do that
by looking at some code.
First, during setup, a view must
register what types it'll accept
by calling
registerForDraggedTypes.
And, in order to accept file
promises, you can use the class
property readableDraggedTypes on
NSFilePromiseReceiver.
Then, when performing the drag
operation, and when enumerating
or draggingItems, you should add
support to
NSFilePromiseReceiver, and make
sure to handle it first.
Because it's more likely to
contain the highest quality
representation.
For each filePromiseReceiver,
you call in the promise, and
when the file is ready, then the
reader block is called on the
provided operationQueue, where
you can handle the file.
It's very important to provide a
background operationQueue to not
block the main trades while
waiting for the file to be
downloaded over-- to be
downloaded, to be written by the
source file.
Because this process can take a
long time, and you don't want--
especially if the file needs to
be downloaded over a slow
network.
So, for a better user
experience, you need to display
a loading activity, and when the
file is ready, then you can
replace the UI with the real
content.
Here you will see an example of
Mail, showing a placeholder UI
while waiting for the image to
be downloaded via file promises.
So, that was receiving file
promises.
Now, let's see how we can
provide file promises.
And, you should consider
implementing this in your app,
if the data you want to send
over drag and drop does not
exist yet on disk.
So, let's have a look on how we
can do that by looking at the
API.
First, you will need to create
an instance of
NSFilePromiseProvider.
You should create one instance
for each promised file.
And, before writing the
filePromiseProvider object to
the pasteboard, it must contain
a file type, and a delegate.
These delegate is doing the
heavy lifting of writing files
to disk.
There are only three methods for
NSFilePromiseProvider delegates.
The first one is called by the
drag destination, and returns
the file name, not the full
path.
The second one return an
operationQueue, and when-- an
operationQueue where the file,
it will be written.
And, we highly recommend that
you implement this optional
method, and provide a background
operationQueue, since otherwise,
the main queue will be used
otherwise.
And, finally, writePromise to
file is called, asking you to
write file to disk.
And, when you are done, remember
to call the completion handler.
So, that was receiving file
promises-- providing file
promises, and we have covered
the API to-- that you can use to
adapt your app to receive and
provide file promises.
And now, I'd like to invite
Joachim on stage to show you all
of that in a sample app.
Thank you.
Joachim?
[ Applause ]
>> Thank you.
Good morning.
We're going to look at a simple
beam generator app, which has a
few issues with drag and drop.
And we're going to dive into
Xcode, and try to fix those
issues together.
So, here is my simple app.
And, I can just grab a file I
have here on desktop, and drop
it onto its window.
Just like that.
On the upper right, I have a
little button, where I can add a
text box, and I can add some
text to my image.
I can select it, move it around,
and when I'm ready, I can send
it over iMessage, for example.
So, while this image is great, I
really want to use an image I
found on the web.
So, let's open up Safari, and
try to drag this image onto our
app.
Unfortunately, this doesn't
work.
So, let's jump to Xcode, and see
if we can fix that.
Here I am in the main
viewController of our app.
And, we going to jump straight
to view [inaudible].
And, have a look at what's going
on.
Right here, we're only
registering for file URL's.
So, let's fix that, and register
for FilePromiseReceiver as well.
Next, when handling the drag
operation, we need to take care
of FilePromiseReceiver as well.
So. Let's jump to the next
method right here.
So, here is the list of
supported classes, and we only
have NSURL.
So, we're going to add at the
first index,
NSFilePromiseReceiver.
In the enumeration method here,
we're going to add a new case.
Where we're going to receive the
promised files, and if we get a
file URL, we're going to call
the exact same method we will
have called if we had just
gotten the file URL over drag
and drop.
So, let's see if it works.
So, here's the little app.
Let's go back to Safari, take
the image, and drop it onto the
window.
[ Applause ]
So, that was pretty easy.
I only had to modify two
existing methods in order to
accept file promises.
Now. Let's add some text.
Like this.
And, wouldn't it be great if I'm
not quite ready to send out my
meme to somebody, if I could
just drag it onto the desktop to
save it as an image file?
As you can see, this doesn't
work.
So, let's jump back to Xcode and
fix this as well.
At the bottom of the class, we
have this method which returns
an object conforming to
NSPasteboardWriting, and as you
can see, we're just returning a
simple NSImage here.
So, we're going to replace it
with a filePromiseProvider.
So, we're creating a file
provider, and we're going to
provide a JPEG image.
And, we're going to use this
userInfo property to store the
snapshotItem, which we're going
to use later to write out the
file to disk.
As a next step, let's conform to
NSFilePromiseProviderDelegate.
Just like this.
Jump back down.
And, implement the three
delegate methods you heard about
just before.
The first one returns the file
name.
For simplicity, we're just
returning a static file name
here.
The second one returns an
operationQueue.
And, we happen to have one right
here.
And, the third method is
actually going to write the file
to disk.
So, what's happening here, is
that we're getting the
snapshotObject out of the
filePromiseProvider object, and
we're going to use its JPEG
representation to write the file
to disk.
So, let's have a look.
This time, we're going to jump
to Photos.
Going to take this image, drag
it into our app, add some text.
And, drop it onto the Finder
desktop.
And, here we are.
[ Applause ]
So, if you just follow the
simple steps I just showed you,
you can add support for file
promises to your app, and
improve the user experience.
And, with that, I'd like to hand
it back to Eric.
Thank you.
[ Applause ]
>> Great stuff.
Great stuff.
I hope you really enjoyed that.
I just have a few words, just in
summary.
To, kind of, tie this all
together.
And, I'll start by saying, you
know, photos really, really
matter to all of us.
Right? Millions and millions of
people around the world take
millions of photos every day.
And, all of those photos find a
home inside the Photos app.
But, they're nothing if they
can't be shared.
If they can't be shown.
If they can't be preserved into
beautiful keepsakes.
And so, all of these people
capturing all of these images,
they rely on you, the developer,
to create amazing apps that let
them do creative things with
those images.
So, if you only remember two
things from this session today,
please remember this.
Support file promises.
You saw it's really easy to have
some very immediate interaction
with the Photos app.
Please, please do that.
And, secondly, we strongly
encourage you to explore all of
the API that's available to you
in the project extension
experience within Photos.
Going back to what I said at the
beginning, this is a blossoming
ecosystem.
And, there's a tremendous
opportunity to embrace it.
And, collectively for us to
create a better Photos
experience for everybody.
So, with that, please join us in
the lab this afternoon.
We have tons of engineers on
staff to help you with any of
your PhotoKit questions, Photos
extensions, be it iOS, macOS, we
really look forward to talking
to you there.
And, enjoy the rest of your
WWDC.
Thank you.
[ Applause ]