# ``
Simon Pieters
Opera `, 2007
```js
play()
pause()
stop()
```
```html
...
HTML5 in BBC News
```
???
In February 2007, Anne van Kesteren from Opera sent an email to the WHATWG mailing list with a proposal for a `video` element.
https://lists.w3.org/Archives/Public/public-whatwg-archive/2007Feb/0487.html
In March 2007, Maciej Stachowiak from Apple also sent an email to the WHATWG mailing list with a more elaborate proposal for a `video` element.
https://lists.w3.org/Archives/Public/public-whatwg-archive/2007Mar/0662.html
A few weeks later Ian Hickson added a `video` element to the HTML spec.
---
class: light
## Codec negotiation
```html
```
???
At the same time he also added a `source` element, to do codec negotiation.
Only `src` and `type` attributes.
https://github.com/whatwg/html/commit/845c40c8035ddccb20301433b2f1225a1203e4de
https://github.com/whatwg/html/commit/61af426a3ae6d68ffa00505b6883cbf870481b52
A video file usually has a video track and an audio track, that are encoded in some codec,
and then put in a container format.
Different browsers support different formats, and so we needed a way to let the browser load the file it supports.
This example uses H.264 constrained baseline video codec, Low-Complexity AAC audio codec,
in MP4 container.
And Theora and vorbis codecs in Ogg container.
---
class: light
## First proposal for ``: 2011
```html
```
```html
```
???
In December 2011, my colleage Bruce Lawson wrote a blog post in which he proposes the `picture` element
together with the `source` element and a "fallback" `img` element.
http://www.brucelawson.co.uk/2011/notes-on-adaptive-images-yet-again/
This was not actually the first proposal for `picture`,
that had been discussed a bit on the mailing list,
but I think it was the first proposal to re-use `source` in `picture`.
As you can see, his example uses `alt` on the `picture` tag.
So in this proposal `picture` was a new `img`,
and once browsers supported it you would not need the fallback anymore.
---
class: light
## First `srcset` proposal: 2012
```css
selector {
background: image-set(url(foo-lowres.png) 1x,
url(foo-highres.png) 2x) center;
}
```
```html
... ... ...
```
```html
```
???
In November 2013, on my birthday, I propose a new approach that supports all the features and avoids
the problems of introducing a new element that renders an image, taking inspiration from the
`datalist` and `input` elements.
https://lists.w3.org/Archives/Public/public-whatwg-archive/2013Nov/0256.html
The difference is that it is the `img` element that is the **active element** instead, so that all
functionality it has does not need to be re-implemented and re-tested for a new element.
This may seem like a minor difference but it turned out to be the last hurdle to resolve before
all browsers could agree to implement `picture`.
Then **Tab Atkins** from Google wrote up a spec for the proposal, and a bit later I took over editing of the spec.
Yoav Weiss who has been involved in this from the start then set up an indiegogo campain to fund
implementing the `picture` element in Blink and, as a stretch goal, in WebKit,
and then he implemented it in Blink and implemented at least the "new" `srcset` and `sizes` in WebKit.
Excellent work Yoav!
---
# Processing model
.scatter[
media element load algorithm
resource selection algorithm
resource fetch algorithm
update the image data
select an image source
parse a srcset attribute
parse a sizes attribute
react to changes in the environment
]
???
Media elements and the `picture` element have different processing models.
I will highlight a few key differences.
---
class: light
## `` loading example
```html
```
???
Media elements try each `source` element in order until it finds one that can play.
It carries on even if it tried downloading one resource and it failed in some way.
If it **reaches the end** of the list, the algorithm is just waiting for another `source` to be inserted.
It never gives up. When it has found a resource that it can play, it sticks with it until you change `src` or call `load()`.
---
class: light
## `` loading example
```html
```
???
The `picture` element only looks at the markup and chooses a candidate in one tick and sticks with it,
even if it turned out to fail when downloaded.
So in this case it skips the first two `source` elements because the `type` is not supported
and `media` does not match, and the third `source` is selected. It will then render a broken image.
---
class: light
## Responsive(less)ness
```html
```
???
`video` is not responsive to environment changes.
For instance if you full-screen the video, the resource selection algorithm will not re-run.
`picture` *is* responsive to environment changes.
If you resize your window or rotate your device, the update the image data algorithm re-runs.
---
class: light
## DOM mutations
```html
```
```js
video.onended = () => {
for (const s of video.querySelectorAll('source'))
s.src = s.getAttribute('src').replace(/^[^\.]+/, 'ads');
video.load(); // need this
}
```
???
For `video`, DOM mutations to the `source` elements will not trigger a new load.
Need to call `load()` method explicitly.
In this example I listen for the `ended` event and rewrite the `src` attributes on each `source` element.
Then call `load()`. Without `load()` it still shows the old video.
But if you change `src` attribute on the `video` element itself, then that does trigger a new load.
For `picture`, DOM mutations to the `source` elements will trigger a new load.
There is no `load()` method on `img`.
---
class: light
## `` is a dummy container
```css
video, picture { border:10px hotpink solid }
```
.example-border[
```
.footnote[Illustration from WHATWG HTML spec]
???
`picture`'s `source` has the `media` attribute though, for art direction.
The value is a regular media query, you can use anything that you can use in @media in CSS.
For example you can use `orientation`, `aspect-ratio`, `resolution`...
---
# `srcset` attribute
???
For `source` in a `video`, you can only use the `src` attribute.
For `source` in a `picture`, you can only use the `srcset` attribute.
---
class: light
## DPI-based selection
```html
1x
3x
.footnote[Illustration from WHATWG HTML spec]
???
`srcset` currently has two possible descriptors, `x` and `w`.
`x` is used for device-pixel-ratio based selection. If the screen has more pixels per inch, you get a bigger image.
You can use any order you like in `srcset`.
When using `x`, the `src` attribute is also a candidate, and acts like `1x`.
---
class: light
## Syntax of `srcset`
```html
```
???
The syntax for `srcset` is pretty simple, you have a comma-separated list of URL and `w` or `x` descriptor pairs.
So to parse it you split on commas, then split on spaces and you're done, right?
---
class: light
## Parsing `srcset`
```html
```
???
Nope. URLs can contain commas. For instance a data: URL always contains a comma.
We want to be able to introduce new, more complex, descriptors in the future, that could maybe
themselves contain commas and spaces.
To support these cases the parser is fairly complicated and has a state machine to basically tokenize the descriptors.
Tests http://w3c-test.org/html/semantics/embedded-content/the-img-element/srcset/parse-a-srcset-attribute.html
---
# `sizes` attribute
???
The `sizes` attribute is used together with the `w` descriptor in `srcset`.
It informs the browser how much big you want the image to be in the page layout.
This is necessary because when the browser wants to start downloading an image,
it has not yet downloaded and parsed your CSS, so it doesn't know the layout.
---
class: light
## Preload scanner
```html
1 column
2 columns
3 columns
.footnote[Illustration from WHATWG HTML spec]
???
The `w` descriptor in `srcset` describes the pixel width of the image resource.
When using `w`, the `src` attribute is *not* added as a candidate.
Repeat the `src` URL as the first item in `srcset` and give it an appropriate `w` descriptor.
The `sizes` attribute lets you describe your layout breakpoints and the size of the image for each layout,
and let the browser take that into account *and* the user's DPI, to choose an appropriate image from `srcset`.
Trying to do the same thing with media queries is much more verbose and brittle,
and requires the web developer to do the calculations instead of the browser.
---
class: light
## Syntax of `sizes`
```html
```
.label[
✗
]
```html
```
???
In the `sizes` attribute you first put a comma-separated list of media condition and CSS length pairs,
and end with a lone CSS length that will be used if none of the conditions match.
The `sizes` attribute uses CSS syntax, so you can use `calc()` and any CSS units you like.
But percentages can't be used because you would expect it to be relative the containing block,
but the layout is not known yet so that doesn't work. `em` and `rem` work the same as they do for
media queries -- relative to the default font size, not affected by the computed `font-size`.
Contrary to the `media` attribute, `sizes` uses a **media condition**, not a media query.
The difference is that in a media condition you can't use keywords like "screen", "print", etc.
The items in `sizes` are evaluated in markup order.
If you get the order backwards then the first item will always be selected.
---
class: light
## Parsing of `sizes`
```html
```
???
Since `sizes` uses CSS syntax, if you want to parse the value yourself, in order to be correct
you really need a CSS parser. Work is ongoing in Houdini to expose an API to get access to the CSS
parser from JS, but that's not usable yet.
The CSS syntax handles things like nested blocks and escaping of characters.
In this example the first block is entirely ignored, along with the 100px.
The 200px with the backslash is valid so that is picked up and the parser stops.
The 300px at the end is just ignored since the parser has already found a value.
The reason this is more forgiving than just ignoring the entire attribute if there is an error
is mainly to be compatible with future additions. We want to be able to add new things here and
still have a way to give a fallback value for today's browsers.
Tests http://w3c-test.org/html/semantics/embedded-content/the-img-element/sizes/parse-a-sizes-attribute.html
---
# `type` attribute
???
`type` can be used for both `video` `source` and `picture` `source`.
---
class: light
## Image format
```html
```
.footnote[See http://flif.info]
???
The `type` attribute is basically the same for video and image.
Except images usually don't have a `codecs` parameter.
This makes it possible to use new image formats on the Web in `img` even though it's not universally supported.
PNG took a long time to get widely used because so long as the browser support wasn't there across the board,
it made more sense to just use GIF.
---
# Double downloads
???
Let's talk about double downloads. What are they?
It's when the browser for some reason starts two requests but you had expected only one.
Today it shouldn't happen but due to bugs it might happen anyway.
---
class: light
## Changing `src` with script
```html
```
---
class: light
## Correct answer:
```html
```
???
The source element disappears.
These browsers support `video` but not `picture`.
They ignore `source` tags that appear in an unexpected context.
---
class: light
## Workaround for IE9
```html
```
.footnote[See http://scottjehl.github.io/picturefill/]
???
For IE9 you can use conditional comments and a video element. The picturefill polyfill picks it up.
For Android 2.3 there is no workaround. You can use `srcset` on `img` as a fallback.
---
# Complex example
---
class: light
## Height and width-constrained
```html
```
???
In February 2012, Edward O'Connor from Apple proposes `image-set()` for "retina" images in CSS
to the www-style mailing list, and notes that the same trick can be used in HTML `img` as well.
http://lists.w3.org/Archives/Public/www-style/2012Feb/1103.html
In May 2012, Ian Hickson adds the `srcset` attribute to the HTML spec.
https://lists.w3.org/Archives/Public/public-whatwg-archive/2012May/0247.html
The `w` and `h` descriptors you see here are not the same as the `w` descriptor that is implemented today.
The original ones were more like media queries, for art direction.
---
class: src-n
## Moar responsive image proposals
???
There are long discussions about `srcset` vs. `picture`.
It goes back and forth and around in circles and new proposals keep popping up every few months.
I try to explain that the **asynchronous loading model `video` has is complex** to implement and we shouldn't repeat it.
Still today no browser passes the tests I wrote for this when we implemented `video` in Presto.
http://w3c-test.org/html/semantics/embedded-content/media-elements/loading-the-media-resource/
I suggested that **using an attribute**, or multiple attributes, would be easier to implement,
because there are fewer weird cases to consider (e.g. no DOM mutations during parsing).
Kornel Lesiński then shows that it is possible to design a synchronous resource selection algorithm
even though elements are used, that avoids some of `video`s implementation complexity.
---
class: light
## The proposal that stuck: 2013
```html
```
```html