Brian Kardell: All right. Hi, I'm Brian Kardell. I'm a developer advocate at Igalia.
Eric Meyer: And I'm Eric Meyer, also developer advocate at Igalia. We're going to be talking about Web and history and things that have to be in the shadows, and we have people to talk with about it.
Jeremy Keith: Yeah. Hi, my name is Jeremy Keith. I'm one of the founders of Clearleft, a design agency in Brighton and the UK. I've been making websites for quite a while.
Jay Hoffmann: And I'm Jay Hoffmann. I'm director of development at Reactive. I also have run a newsletter and blog called 'The History of the Web' for about a decade now. And so I thought it would be fun to chat to all of these folks about specifically different APIs and things throughout all web history, starting with one of maybe the more elusive ones, which is Shadow DOM. That will be our topic for today. You all ready to talk about Shadow DOM?
Eric Meyer: I mean, ready as I'll ever be.
Brian Kardell: Where do you want to start?
Jay Hoffmann: I think if we really want to start with Shadow DOM, we have to take it very far back, potentially to the early days of Netscape Navigator and talking through... I mean, there were various attempts at essentially what I think is the primary purpose of Shadow DOM, which is encapsulation. Maybe that's starting from an incorrect premise, but certainly there were a lot of attempts at this encapsulation early on starting in the mid '90s with things like Java applets. Maybe we can start with just, is that the correct kind of framing for this, or what other motivations were there when browsers started to think about this?
Jeremy Keith: Yeah, I mean, I feel like there's always been this quest for components of some kind of various levels of isolation from the surrounding document. I mean, going way back, you had frame sets. Not even iframes, but like frames, which in theory are isolated. There was a certain amount of communication between them. You could click a link in one frame and it would open in another frame, but they were effectively standalone HTML documents. That's probably the first inkling I think of that idea that there's a whole page and within that page, there's these standalone components. But they didn't last. Frames were a bad idea.
Eric Meyer: Why?
Jeremy Keith: I mean, the accessibility implications, for one, were pretty nasty. And then just usability. As soon as you've got this idea of some of the navigations would take you away from the whole frame set. Some of the navigations would open or have an effect in a different frame than the one you clicked on. Some of the navigations would happen in the frame you're currently in. Then immediately throws up the question like, 'Okay, then what does the back button do now? How does backwards and forwards work in this world?' The answer was, I think, we have no idea. Let's kill this whole thing.
Brian Kardell: But I mean, we do still have iframes, and iframes have the same problem. They're still popular. But I would like to say that maybe even before Netscape that you could take this back to Pei Wei with ViolaWWW. That had something like embeddable frames, but like the idea there was also close to what became Java. You had a universal runtime, and you could bundle these things in one another. Which the idea there also, I think, had to do with encapsulation. Yeah, I mean, I think it's just natural in programming to want to do some kind of encapsulation, right?
Jeremy Keith: Yeah, it's the same thing that effectively you got with Flash later on, or like you said, Java applets. This thing is a standalone thing, but you can also put this thing into the broader context of a webpage. People did use Flash in both ways. Sometimes people did have just little bits of Flash within mostly a webpage. Sometimes the entire webpage is mostly Flash. It allowed for both. It was interesting, actually. I remember when Flash was really a big part to the Web. At this point, frames and frame sets were long gone. That was ancient history. But all of these questions about accessibility and navigation and the backward and forwards button, they all resurfaced again. They came back to life like, 'Oh, now that I've built effectively all my navigation within this island of Flash, now the back button doesn't work the way people expect.' It's just interesting to see that these usability questions just kept coming back as soon as something stopped being a little island of encapsulated something to being most of the webpage, most of the website.
Jay Hoffmann: I think what's interesting is that as you had things like Java applets and Flash beginning to develop, you simultaneously had a lot of efforts to standardize something that would be closer to what people actually needed. An example being HTML components, which date back to, I think, the mid to late '90s and XBL, which I'm sure we will talk about. I'm curious from your all perspective, I mean, is it that because they were not standardized that something like Java applets and Flash could have their day and build and iterate on features that they needed? What were the reasons basically for its success? But then also for its eventual collapse in all of these cycles that you're talking about, Jeremy, it go up and down.
Jeremy Keith: Yeah, in my opinion, they absolutely were able to drive innovation, which I hate that word. It's a much abused word in the Web. But the fact that it wasn't a standardized thing, I'm talking about Flash now, but it just could apply to other things as well, when they could just push forward and just launch features and get things out there. That was, I think, ultimately beneficial for the standards process because developers, users, got to try stuff out and say, 'Oh, I like this. Oh, I don't like this. ' Can we keep animation? Can we keep video streaming on the Web? Can we keep audio happening on the Web? That's good. But in this context of one company proprietary, inevitably it always ends up eating itself. It's usually down to business reasons rather than any technical reasons. Sure enough, that's, I think, what happened with Flash. But I also think, yeah, Flash was a trailblazer in showing us and showing standards bodies people want this. People like this. Let's find a way to do this.
Eric Meyer: Yeah, that's the role that I think pre-processors and some frameworks fill today. A lot of things in CSS came from SaaS because people were able to try things out in SaaS. And then they have made their way into CSS either quickly or slowly, depends on the thing. I'm sure there are other things that have been done in SaaS that no one has ever asked to be in CSS because six people wanted it. Everyone else said, 'This is terrible.'
Brian Kardell: There's an interesting observation that I have here. We talked a little bit about, we mentioned Flash, we mentioned Java. I don't know if we want to talk more about Java. I also mentioned Pei Wei's idea. But those were a webpage, some HTML document that you would then create a portal into a completely different thing, right? You had web stuff and then you had this other stuff that was embeddable in the Web. And then there was both HotJava and Pei Wei's thing, where you had the virtual machine and you had the web browser written in the virtual machine, right? And then within the page, you could have more virtual machine. Do you know what I mean? It was your building of itself, but the of itself wasn't of the Web. It was of whatever that runtime was, right? I think maybe later on, the closest we came to this was Flex in Flash. Flash came up with its own answer, so there were, at the time, I think... I don't know if we want to get to this yet, but we wound up with three major competing ideas. There was XUL, XAML, and Flex, and they were all very, very similar in a way.
Jeremy Keith: I want to hear more about this, about this. Jay, you went splunking and found out about the origins of a web component.
Jay Hoffmann: Yeah, absolutely. Yeah, first of all, just by way of context, ViolaWWW, for those that don't know, that was developed in 1992, right? This is right alongside what Tim Berners-Lee is doing, essentially, right? We are talking about going back to the beginning of the Web and flashing all the way forward to the end of it, which means there were a lot of stops and starts. I do think it's going to be worth digging into these terms, Brian, to try to understand them and how they all forked in different directions. If you dig into the drafts of the W3C and you look through the specifications, you may come across something that's called HTML components. Which I don't exactly know when it began, but I know that it was in a more mature form by the end of 1998. Imagine that this is all happening. There are all these kind of different attempts at encapsulation and components and certain people, including Chris Wilson, participated in creating HTML components. Which very interesting specification for those that would be interested because it really does mirror the, at least from a syntax perspective, what we all have now more or less become familiar with is the idea of single file components, where you can have your behaviors and your styles and your, well, styles in a very elementary form, and your markup all in one place. And so I think that there were attempts to try to find a way to address this through standards. I think the earliest that I could find was this idea of HTML components, which in some ways I think informs what would come next. I'm not sure if anybody ever really got a chance to use HTML components, unfortunately, because I think it really just emerged as a standard. I am sure there are test cases somewhere that we could find at some point, but really no further adoption than that. Maybe specifically because all these other alternatives were starting to take off by the time we get to, let's say, the late '90s, early 2000s.
Jeremy Keith: Had some interesting ideas in it, like the mechanism by which you pointed at the thing you wanted to include, was it the way you pointed at this other self-contained component. Am I right in thinking it was like pointing at a style sheet, but for a component?
Jay Hoffmann: Yes. I think it was designed to be as HTML was at the time and continues to be, obviously in a sort of declarative way, and then even tying behaviors to declarative statements. Which the separation of concerns that evolved with WASP and web standards movement and things like that probably would've taken this in a different direction anyway at that point. But yeah, absolutely. I think it was very much about tying behaviors to markup, which is such a fundamental piece of the Web. I mean, you can maybe argue that when Backbone.js came out, that's truly what it was trying to address. It's just very interesting to see that there were, as you said, Jeremy, stops and starts to all of this as we continued through iterating on the ideas. So I think it’s worth now turning to — you know, maybe this would be hard to understand for people that are just starting now, but talking a little bit about XML, which would lead us to XUL, which was an outgrowth of that, which was an attempt at components in a different way at Mozilla. The framing I would give, and you all have, I'm sure, lots of perspectives to bring to XML in general, but the perspective I would bring is that XML became pretty popular at the W3C. There was an idea that HTML would evolve into a version of itself that embraced XML with much stricter rules around how markup was formatted in an attempt to create, maybe 15 years too early, an internet of things in some way. Where if the underlying document could be semantic enough, that machines could understand the markup as well as humans could, and then you could do all sorts of interesting things. This was eventually abandoned, but that is what was in the air. This is why I think XML became the next place where you see attempts and encapsulation and components happen. Yeah, let's talk about XUL and XML and how those two things combined to next, I would think.
Jeremy Keith: Well, I remember first coming across XML in the late '90s and it was in the ascendant. On the one hand, it was a formatting format for structuring data. It made sense to me. I could look at it and it would make sense. Like, 'Oh yeah, I get it. You can structure anything any way you want.' On the other hand, it was also, it was a way of life. I remember it really did feel like, have you let XML into your life yet? It was our Lord and savior XML. It really seemed like everything was going to be XML, which explains why, well, let's turn HTML into XML. We're halfway there already with the ML part. Let's go the whole hawk. But everything, if it could be XML, it was going to be XML. That was my memory, that it was not so much about the format, it was the culture. Brian, you remember this?
Brian Kardell: I 100% remember that. I think part of it comes out of the fact that HTML has its origins in SGML, which had been around for 30 years before that. More or less, HTML, just a whole bunch of existing thousands and thousands of documents were readable in your browser already on the day that Tim made the browser. If you knew how to find them, you could open them because it worked with this forward compatible parsing and partial understanding. You might only see some headings and paragraphs, and that's all you can make sense of. The rest, you just don't pay attention to the tags. You just still display the text. When HTML came along, it was itself not really SGML. SGML had standards, like ISO standards. There were many, many, many ISO standards, groups, and there were these two competing formats, like SGML and EDI in terms of trading data, right? The people who saw that, saw the Web taking off and they were like, 'Hey, wait a minute. I'm glad you're on board, but let us clue you into all the things that you're missing.' And so there was this opportunity to loop in all the SGML people when they started W3C. They did. They thought, well, we could fix SGML and make XML. And so the whole world of industry was behind that because it already had momentum on that side and now it had this excitement momentum. I think that like you said, Jeremy, it became like... I don't know about a religion, but people have compared to the high priests of XML or something, but you got to do everything in XML. I'm surprised actually that I think that if there was a world in a race that was slightly different, even CSS would have been expressed in XML.
Eric Meyer: Well, they tried. That's accessible style sheets formatting objects. That was an attempt to make CSS an XML. It was incredibly baroque in both ways you can hear that word. Just imagine taking CSS rules, turning them into XML, but really, really complicated XML. It was tried. Pretty much, like Jeremy said, everything was going to be in XML.
Jeremy Keith: Even though that reality didn't come to pass, everything being XML, I feel like maybe you can draw a bit of a line from that time period and today. Not so much with Shadow DOM, but with the other part of web components, which is custom elements. The X in XML was for extensible. The idea was, okay, you want to extend HTML and come up with your own elements. Yes, okay, now what do they do? How do you give them behaviors and all this other stuff and encapsulate what's inside those new elements you just made up? But that idea, extensible markup, that's what we ended up getting with custom elements. It looks different because you got to have your dash and all of that, but it is there to solve that problem. It's like HTML doesn't have enough elements. I want to make my own elements. All right, here you go. It's not using XML, it's using custom elements, but tomato, tomato.
Brian Kardell: I mean, this is part of why Tim reasoned that forward compatible parsing was a good idea too, right? People are going to want to make their own tags. You can't just stop parsing, right? People in this sense were probably mainly still browsers when he was thinking about it because there were like 35 browsers back then. It was a hobby thing that people would build, but he thought people are going to want to build their own elements. And then eventually some of them will get popular and copied, and there you go.
Eric Meyer: Yeah, I remember that promise of XHTML actually was that, oh, well, it will be extensible. We won't be limited to just the elements that are defined. Actually, in the first edition of CSS, the definitive guide, some time is spent on why CSS will be important when you can invent your own elements. Not a lot of time, but a little, because right around 1999, 2000, that was what was being promised. Yeah, in custom elements, we can do that. Yeah, there were a lot of people who were really excited about, oh, I can invent my own whatever element that will capture structures that HTML doesn't. We can have a poem element with stanza elements and whatever. Now, yes, we can do that now. Because CSS is language agnostic in a sense, it will style any markup that you send that you tell it to style. We have that. Yeah, it's cool to see that part of the promise come to pass.
Jay Hoffmann: There was something else happening around this time, which I think really informs where Shadow DOM or web components goes next, which is that AOL purchases Netscape for a very large sum of money. As part of that, they're actually starting to abandon the browser game and moving it into an open-source foundation, or what will eventually be an open-source foundation known as Mozilla. They start working on a browser called Firefox. As they're working on taking Navigator, what was Netscape Navigator and transforming it into Firefox, they start to take a look at the underlying structure of the browser. Now XML is very available to them, and so what they develop is something that's called XUL, which is XML user interface. They really use this as a way... There's all these sorts of components that live outside, like in the chrome of the browser, outside of the webpage itself that they wanted to be able to develop using some sort of standards and in a way that was consistent across operating systems. And so that's where XUL starts to come from. That's probably, as you were saying, Jeremy, one of the first instances that we have of custom elements, like this idea that we're going to build custom elements with complete encapsulation. This is actually, as a side note, Brian and I started talking about XUL six years ago, right? It's also just something that I've been interested in and just remains underneath a lot of the work that I've done in looking into web components, but it's very... It's a hidden API or there's something that was really being developed by Mozilla specifically.
Brian Kardell: Just a fun little footnote that you said extensible user interface, but there's cleverness in here that this is actually a reference to the movie Ghostbusters. The XML namespace is mozilla.org/keymasters/gatekeeper/there.is.onlyxul.
Eric Meyer: Yeah, so XUL is pronounced Zuul typically.
Brian Kardell: Yeah.
Jeremy Keith: Am I right in remembering that this was proposals the way you would make browser extensions, that it was the language you'd built them in? Or have I got that wrong?
Brian Kardell: No, they used it like they dogfooded their own thing. There's a bit of this that is part of why I made this observation early on, which is about like, what thing are you building out of what's your mental model? There was this idea at the time that HTML is good. It was cute for documents, but it would be great if we had something like that for applications. At the time, there was this big idea around like a portable virtual machine, one language you could run. Could we find a way to build something that was like applications with this XML described user interface and a virtual machine? That's where you get HotJava. It's also where we spun out these ideas. Microsoft comes up with XUL and... Not with XUL, with XAML, and Silverlight is their engine. Flash has their own engine and they create Flex. Mozilla creates XUL trying to keep it standard. Both Flash and Microsoft were betting they could own the second Web, the application Web. Mozilla wanted to do something in a standards body, so they did this thing. They tried to take the ideas that they saw that were being discussed with HTML components, the good ideas, the not so good ideas, learn something from it, and try to standardize XUL. A lot of Firefox was actually built out of XUL components, like the toolbars, the menus, all that kind of stuff.
Eric Meyer: Yeah, I mean, XUL was for describing visual components of Web. The browser Chrome could be described in XUL. You could add a browser toolbar by describing it in XUL. I mean, that part of web components, yes. Or Web, sorry, of that part of browser extensions, yes. You also needed XBL, I believe, in order to change the behavior. There was XUL and XBL, which didn't-
Jay Hoffmann: I think XBL was an outgrowth of that, of XUL. And then as Brian said, Microsoft was trying something relatively similar with XML, at least as the backing, as XAML. Yeah, I am curious from your all perspective, what are the good ideas here? What was interesting about XBL? What was interesting about XAML and Flex and all of that that would later inform some of the features that we have today?
Jeremy Keith: I guess it's interesting that this has all been about encapsulation... Well, two things. Encapsulation one, extensibility the other. That's already come up talking about XML in general. These are all XML based things. There's this melding of extensibility and encapsulation that seems to be a thing people want. It's interesting how you described it there, Brian. All these companies are trying to be the ones who are going to win this battle for this combined extensibility and encapsulation. And then I mean, maybe I remember the history wrong, but this would have been done around the time that a bunch of people at some browsers like Opera took a look and said, 'Wait a minute, HTML isn't dead. It's just resting. We can revive it.' As far as I understood it, the idea wasn't to chase the dream of encapsulation and extensibility in its abstract form. It was, why don't we just give people what they mostly want? People are crying out for video and audio and some more form controls and stuff like that, so what if we work on that? I know things got very messy there for a while with the what working group and the W3C, but leaving all that aside or we'd be here all day. I think the interesting thing was what became HTML5 was not about extensibility and encapsulation in an abstract way. It was about extensibility in a very targeted way. We're going to extend the input element. We're going to extend HTML. We're going to add some new elements. I feel like maybe the narrative changed there as well. It was almost like, oh, we tried the infinite extensibility thing in the XML era. Look, we did XUL, we did XAML, we did all these things. It didn't work out. And then when we went for limited extensibility, lets us all agree on the 80% use case. Oh, that worked out well because we got HTML5 and that turned out to be a great success. Maybe there was almost like a bit of a detour there where infinite extensibility just took a backseat for a while. Ah, we tried it. It was great in theory, not so good in practice. It took a backseat until HTML, it felt like got back to a stage where it's like, okay, it's time to start talking about infinite extensibility once again.
Brian Kardell: Yeah. I mean, the split that happened in W3C was over curiously enough what to do with compound documents and web applications. There was a group that was working a lot on the extensibility, and specifically like Xforms was a big part of that. The sense was there was a workshop and people presented different position papers. Yeah, there was a sense that one argument was, look, making the Web break the way that we're doing is like not helping anything. Maybe we can get to this amazing XML future, maybe. I'm not saying we can't. Just maybe there isn't such a clean break between application and document. Maybe we can take the good parts of what we learned in those other things and put them into a more finite answer in HTML. And so they took a lot of the things from Xforms that seemed like good ideas and tried to bring them into HTML, and to do it quickly by web standards, which I think meant about a decade is what it took.
Jeremy Keith: Yeah, and my memory is that after that decade is when then we started to get the first murmurings of Shadow DOM. It was like-
Brian Kardell: That's correct, yeah.
Jeremy Keith: We figured all that stuff out and got it to a good enough state. It's like, 'All right, time to pick back up that thread.' I don't know if anyone else remembers it like that, but maybe I'm applying it in hindsight.
Brian Kardell: No, that's pretty accurate. It is interesting though that Hixie, Ian Hickson, who was the editor of HTML, also was sending things around about XBL because he was keen on what to do with XBL. Its charter expired. We had XBL1, XBL2, like neither of them was actually what Mozilla did. There were things that were problematic and like, what do we do about it? I don't know if, Jay, you want to talk about-
Jay Hoffmann: Yeah, I mean, there was essentially a 10-year reshuffling here, right? Which is funny because I think when you talk about web history, you often are talking around the gap that exists from 2002 or so to like 2010 with the emergence of HTML5. And so I think that there's this reshuffling that happens, and so not much happens in... XBL, as you're saying, is like continues to be developed. It's not like Silverlight's going away or anything. All of this still exists, but there's not a lot of forward progress until we get to, I want to say, 2010, 2011, when this idea is now picked back up. And so a lot of it is it is an outgrowth of people thinking about this for a long time. We are not starting from scratch when we are talking about Shadow DOM. We are starting from various starts and stops and implementations that we've discussed here. And now in 2010 or 2011, somewhere like that, you start to get conversations about how Shadow DOM can be structured. I think a lot of that came from WebKit specifically early on. It's going to get confusing again because we're going to have to talk about the first iteration of Shadow DOM and the second iteration of Shadow DOM, which are somewhat different. Don't worry, we still have plenty of confusion to wade through here, but we did start to get what will eventually become what we're all familiar with as Shadow DOM. And so that's 2011 or so.
Jeremy Keith: Would I be right in thinking this was around the time that we got a manifesto on this idea with the Extensible Web Manifesto, or did that come later? I can't remember when that was first published, but I mean, we're back to that word again, extensible.
Brian Kardell: It came through those discussions, but it came a year or two later. Yeah.
Jeremy Keith: Okay. Yeah, yeah. It felt to me like what it was saying was, 'Okay, HTML5 was a great initiative for the 80% case and we got good enough. Now it's time to tackle the 20%, and here's how we should do it. ' We nailed this manifesto to the church door. This is what we want to do is we want to give developers access to the low level APIs and browsers and let developers figure out what they want. And then we can come along later and standardize the good stuff that comes out of that is my recollection of what the thinking was.
Brian Kardell: Yeah, that's pretty right. I think not just the low level, but medium levels, we talked about how all of these other sort of systems that you would build on have multiple layers. You can look at an architectural diagram of them and layers. You can say, 'At this level, I want to build something on top of that layer.' You couldn't do that with the Web, right? There were not a lot of things you could plug into, exceptionally low level or even something intermediate. The only thing that we had that was like that was the DOM. With just the DOM and some scripting, we had done so much. I mean, polyfills is, I think, how we found our way out of stagnants. We're at this catch 22 where the world was 95% Internet Explorer Six, and that team had been dismissed to go work on Flash and Silverlight or Silverlight and XAML.
Jeremy Keith: Jay, you were saying that there was these different levels of Shadow DOM, like there was a level zero and there was false starts, level one, level two. But in my memory, the way I was sold the origins of Shadow DOM was this is what browsers were doing anyway. The breakthrough came with dev tools. I may be misremembering, but the way it was always explained to me was, 'Hey, did you know if you looked inside a select element, it's got its own little miniature dom in there.' One of the browsers allowed you to do this and dev tools. Now all the browsers allow you to do this, but that was how to explain that this Shadow DOM thing isn't being invented from scratch. It's always been there. Browsers have been doing this the whole time, but now we as developers would get access to that. Which by the way, this was a naive response, but I always found that explanation a bit weird to say like, 'Oh, here's how we'll explain Shadow Tom is. Look at the select element. Now look inside, it's got its own little document inside the opening and closing tag.' It's like, the one thing I know of the select element and most developers know is is it's a real pain in the ass to style a select element. Of all the things to try and sell me on a technology, it's like, you know how to select elements really hard to style, now you'll be able to make things that are really hard to style, too. Which was a naive response, but then also looking at the problems with Shadow DOM today, yeah, I wasn't a million miles off to say the styling issue is an issue.
Eric Meyer: Yeah, no, you were not being naive at all. I feel that you were exactly correct.
Brian Kardell: I think, Jay, this leads into a thing that you want to talk about, about what did we even mean by encapsulation, right?
Jay Hoffmann: Yeah, I think that what is interesting is that after having tried different things for over a decade or maybe 15 years or so, finally I think when Shadow Dam is starting to be talked about in discussion threads, there is this moment where I think especially folks from Apple, I believe, were saying, 'Let's pause for a minute and just talk about what do we actually mean by encapsulation because that is going to be what everything else flows from.' There's a discussion thread about that, about how do we define encapsulation. And then I think, Jeremy, what you're picking up on is then there is what grew out of that was a real attempt to like, 'We need to explain this to people. Because if we're going to get this out there, people need to understand what is happening because it's maybe one of the more advanced things that's ever going to be introduced.' It's not a new element, right? It's something almost entirely new.
Jeremy Keith: It makes sense to define encapsulation because it can mean different things. There's encapsulating styles and that's a whole thing, not having styles leak in or out of some boundary, and are we talking about styles? Are we talking about behavior? Do we want the same rules to apply when we're talking about behaviors when we're talking about styles? It makes a lot of sense to say, 'Okay, you keep using that word. I don't think it means what you think it means when it comes to encapsulation.'
Brian Kardell: Yeah, so Maceij sent this email, it's famous email. Anne van Kesteren wrote about on his blog a couple of times now. It's Wednesday, June 29th, 2011, and he lists five different kinds of encapsulation, actually. Encapsulation against accidental exposure, which is like, you don't want things just willy-nilly leaking out through whatever so that they get easily screwed up, which is basically what we have friendly fire. I wrote a blog post about this called 'Friendly Fire'. Then there's encapsulation against deliberate access, so no API being provided, which lets code outside the component, poke at the internals, that the internals would choose what to expose. That's like today we have like export parts, right? There's inverse encapsulation. There's isolation for security purses. There's inverse isolation for security purposes. Yeah, so Google's original proposal just had a thing called Shadow DOM. It just had a shadow root, and that was it. That was more or less the whole trick, right? There wasn't more than that. If you include a script, it's just also in your document. There's no JavaScript encapsulation. There's not a lot that's special in here. You can walk across that boundary. Apple was really concerned that what they wanted was something like what the browser does. They want to be able to make a component and tell you what you can and can't style because they don't want anybody to break it, right? They don't want anybody to break it. The only way that they can guarantee that is if they control more or less everything about it. This is where we get the closed shadow root, so now we have an open shadow root and a closed shadow root.
Jay Hoffmann: But I think the original definition hinged on that definition number two, which is encapsulation against deliberate access, which is that closed shadow root idea. Which is I think where level zero comes from and potentially where the divergence is between what would eventually become the Shadow DOM we know today is and that first attempt. Does that feel like an accurate way of framing that, Brian?
Brian Kardell: Yeah. Yeah, there were some other things here too, but yeah, more or less, there was this proposal for how you would project things into the Shadow DOM that's different in version zero and version one. But that being an important thing, right? If you want to be able to define something, is it just these islands, or is it these sort of islands with holes in them that you can put things into? That's a thing that's interesting about the Shadow DOM. Previous editions of the Shadow DOM also had multiple shadow roots and very complex inheritance relationships where you could customize something and then I could customize that thing. It's very complicated. I think you could in the original Shadow DOM you could customize an The element. It's interesting when we started these, there was this, even back in maybe 2010, there was this Component Model Use Casts page, it's still out there, maybe we can link to it in the show notes, that was components use cases. I remember Ian Hickson only wanted to do the is syntax. You'd be able to say like, 'Input is something,' and then the idea was that you would then just get accessibility for free. You can see in that component use cases, one of the ideas in there that he explained to me was, well, if you were installing operating system back in '92 or something like that, you would have had some primitive select menu. You would just have your arrow keys and that's what you could do to set the time zone. There would just be like an alphabetical list, and you would up down until you found it. That's it. But now today you have these like nice graphical map things. You can find your city from points, and you don't want your arrow keys to move up and down alphabetically. That would be like really, really strange if you were like Pennsylvania. No, I want to go down because it's Virginia, but then you wind up in like Portugal or something. Very strange geographic leaps. But the challenge with that was and is, this was actually Apple's eventual objection to supporting is at all, is that a select element is not a map, right? Like I say, the keyboard interactions work differently. You have to provide new DOM, and as soon as you provide new DOM, you have to provide new behavior. Part of the reason that I mentioned that though is that like some of those other things that predated this had different splits as well. I think it keys in on this like underlying thing that I stress a lot, which is like, we don't actually know what we're doing right? We're trying a lot of things and there's a like, wouldn't it be great if we just standardize this thing? As soon as we start standardizing this thing, we also figure out something problematic with it or better or whatever. And so you can see these kind of interesting trees. I don't know if you've ever seen those diagrams where they diagram like the evolution of music or something like that. There are interesting evolutions of Shadow DOM-like ideas. I think in XAML there is still a logical tree and a visual tree. Internally in the browser, there's also this kind of thing with the accessibility tree with being the logical tree and the visual tree and how they're related.
Jeremy Keith: There's different things here because the whole discussion around is and extending HTML. Yeah, it is related to Shadow DOM, but really that's more about custom elements. How do you do custom elements? Can you extend HTML? You have to make your own custom element. And then meanwhile, there's Shadow DOM, which is about what happens inside that custom element, the encapsulation part. I think it's good that those two things are separate, that you don't have to do one with... Well, I guess you have to do a custom element in order to do a Shadow DOM, I think.
Brian Kardell: You don't. You don't. That's why I was bringing it up. Yeah, so this create a lot of tension and friction because how does that work? You can create a div. You can go just get a div or a bunch of elements in HTML. You can get maybe 100 of them and just create a Shadow DOM on them. No problem at all. But we can't let you do that on interactive elements, so you can't create a Shadow DOM on interactive elements. In that model, you do need to create a custom element and use composition rather than inheritance.
Jeremy Keith: But I think it is good that they aren't too tightly coupled, these different technologies that all fall under this web components banner. You don't have to use all of them. It's maybe a bit confusing that when the definition of what is a web component is given, it's got all of the possible things listed, like templates and slots and Shadow DOM and custom elements. Maybe that gives the impression that a web component is when you use all of those things. But I really, really like that it doesn't have to be that way. Eric, when you wrote that blog post about using the Light DOM, I'm totally going to use a custom element. I'm not going to touch the Shadow DOM. That got a lot of traction that really resonated. I think the light bulb went off for a lot of people because they looked at Shadow DOM or tried Shadow DOM, and like, 'Oh, this is really not what I was hoping.' Oh, it totally encapsulates things except when it doesn't. It's just so many edge cases-
Eric Meyer: It totally-
Jeremy Keith: And the-
Eric Meyer: Sorry. It's totally encapsulated things, including things I didn't want to be encapsulated.
Jeremy Keith: Exactly. Yeah, yeah. The idea that you could just not use it, and that's a totally valid web component just using custom elements, makes a lot of... Now in other situations, you totally do want to use them. My friend Trys Mudford, he wrote a great blog post about writing a web component where he did need pretty much 100% encapsulation because it was going to be used like an embed. It was going to be used as in like, drop this into your website and it opens a little portal to our website. Like when you were embedding a tweet or a YouTube video, that kind of thing. In that situation, yeah, you're going to want a much bigger level of control than me. When I'm writing web components, I am both the author and the consumer usually, so like my use case is so straightforward. If I don't want to mess with Shadow DOM, I don't have to. Also it's interesting that like if I do come across a situation where I really want to scope these styles, I want to encapsulate some styles, Shadow DOM is one way to do that, but it's not the only way. That's nice. We've got multiple ways of scoping styles and not letting styles leak or letting styles leak depending. I think choice is good. I'm happy with the situation as it is now.
Eric Meyer: I just wish it were easier to pierce that encapsulation layer when you need to. I ran into this just recently. I was using a web component that somebody else wrote to do one of those slide-y comparison things for images where you can... You have two images on top of each other, but as you slide, you reveal one, obscure the other and vice versa. Given the way I was trying to use it, the actual divider with the drag part was not nearly visible enough. I tried to customize it, and I eventually more or less gave up because they had exposed seven custom properties. That was it. That was all you could do unless you could figure out how to modify, and I didn't... Yes, if I dug into the guts of this thing, I could probably have figured out how to do what I wanted to do, but I didn't have time. This was not the centerpiece of a thing I was doing. It was a, I need to put this thing out and it needs to have something like this. I'm going to make it work as best I can.
Jeremy Keith: See, that in itself is interesting because I think if you're going to write a web component that you're then going to put out there for other people to use, I do think you need to approach it with the same level of responsibility as if you're a browser maker and you're about to mint a new HTML element. Maybe not quite that level of thinking through all of the possible implications with accessibility and internationalization and customization, all of those things, but it does tie back. The promise of web components was, 'Hey, you'll be able to create new elements just like a web browser does or has been doing.' But I feel like the counterpoint to that is if you're going to do that and you're going to share these with other people, you need to think as if you're making a web browser.
Eric Meyer: Yeah, and realize that more people than you are going... People are going to have use cases other than yours. I'm not here to say that the person who wrote that web component was engaging in malpractice or whatever. It was frustrating because in the web inspector, if I dug into the Shadow DOM, which it will show me what's in the Shadow DOM, and I could see like this thing here is an SVG, like the arrows. If I could do this thing or if I could style that SVG, okay, that would work. I could put a background image and a border radius on the SVG that contains the two little arrows, and that would be enough. But I couldn't get through that layer enough to make that happen. This is why when I write 'web components', I'm not really writing what web components. I'm writing custom elements because I want all of the guts to be accessible to anyone else who might use them, including future me. Six months from now when I come back to the thing, whatever it is, I'm like, 'Oh yeah, I need to modify this because of reason, reason, reason, reason.' Yeah, I want it all right there. I want the whole thing in the Light DOM. I want all of the CSS to be able to apply. Yes, that does mean sometimes that there's the possibility that the page CSS will affect what's in the thing. But to me, that's okay because I put the thing in that page. It should be affected by the overall page. If I want to not do that, then I just write some CSS and not do it. It's not a huge deal as opposed to the veil of shadows prevents you from advancing. Sorry, it frustrates me.
Brian Kardell: The other kind of friendly fire that they're trying to prevent isn't just CSS, though. It's like DOM selectors where we want to attach events and build new DOM. We select something. It could very well select your guts and your component and respond to your events. And so-
Jeremy Keith: I'm glad that the Shadow DOM exists. I'm probably not going to use it.
Eric Meyer: Yeah, yeah. I'm in the same place.
Jay Hoffmann: What's interesting is web standards people and browser makers and everybody spent however long we've been talking over this chronological period, which is essentially 25 years, about encapsulation. Which brought us to Shadow DOM, which brought us to the tools that we have today. And so I think that there's something interesting about the idea that Shadow DOM becomes the enduring principle by which we keep returning back to, right? Every five to 10 years, we return back to that idea in a new form. Now we have it and it's in 2015 that starts to get implemented in browsers. We have it, which then allows us to start thinking about all of these new problems in much more interesting ways. How great is it that we as web developers can now have a nuanced conversation about what the exact level of encapsulation that we want is such a gift when you think about it in terms of the binary way that this usually turns out.
Eric Meyer: Yeah. I mean, we've had them for 10 years and people are still, like me, complaining about the encapsulation level that was chosen. This is-
Jeremy Keith: But now we have options, right? Now we can do, well, maybe I'll use cascade layers, maybe I'll use scope.
Eric Meyer: True.
Jeremy Keith: At least now we've got other ways of potentially solving our problem, right?
Eric Meyer: I mean, yeah, the encapsulations is, like you say, come into CSS or in some sense is already there. CSS was completely not encapsulatable really within itself, and now you can encapsulate bits of CSS. From other bits of CSS, we've had the Shadow DOM to encapsulate DOM and Markup and CSS. Yeah, I feel like now, you know how you were saying, Jay, you were saying over five to 10 years, we would talk about wanting encapsulation. Now we have encapsulation. I think we're on the 10-year point where maybe it's time to talk about how can we make the encapsulation much more flexible. We have encapsulation, right? But how can we make it so that, as an example, is there a way that I can have a web component where I say the DOM is completely encapsulated, but the CSS is not. Or vice versa, or both or neither. As an example, is there a way that we could do that just simply with attribute values or something? I'm open to suggestions from people who are way smarter than me and have spent lots of time thinking about the actual guts of web browsers. But I think it's time to talk about what kinds of encapsulation do we want, or do we want all of them? Thus, if we want all of them, all of the five types of encapsulation that Maceij described, how do we make those available to authors and developers?
Jeremy Keith: The other cyclical thing I just wanted to briefly mention is that as I remember the idea of the Extensible Web Manifesto was, here we give you the underlying guts of the browser. You go build, extend the Web with these components, here's Shadow DOM, here's custom elements, here's all this stuff. And then after a while, the good ones will come into the fold and become fully-fledged native elements in browsers. I haven't seen that happen, except maybe we're starting to see things like CSS-only carousels, which under the hood are actually like... There's little bits of Shadow DOM happening there. That's what you're actually doing. Maybe a time period has come where you start to look around and go, 'Okay, what's worked and what hasn't?' What's worked maybe does become honest-to-goodness HTML or CSS.
Brian Kardell: I think it's worth remembering too that all these things are being tackled not exclusively. There's lots of other things going on. They're getting fractions of people's time. They're getting fractions of budget. And so you end up inevitably having to draw a box around something and say, 'Let's see if we can ship this part and then deal with the next part once it's shipped.' And so the reason I'm bringing that up is because you can look all the way back at proposals from like 2011, 2012, and they had much more stuff in it that we still don't even have. But we have been getting them. Every few years we figure out a little bit more, we get a little bit more. We get a little bit closer to those original ideas. They're usually not the way that we imagined them. They're usually tweaked and matured in ways that make sense with another seven, eight, 10 years to think about it. But all the way back then, they had effectively declarative Shadow DOM. You had how we're going to use the template element to create these things and how you could maybe do it declaratively. Now we have declarative Shadow DOM, but it's the thing that we could ship. It doesn't have... They had templating. They had a proposal for templating. They had a proposal for attaching bindings that was very much like .htc called behaviors. All of these things. And so currently there are, and there have been for several years now. This is five years, so we're hitting this point. It's like this has been developing for six, seven years. There's discussions about open stylable, which is like the thing that you're asking about. Is there are a way to do this? How would I do it? A lot of that has to do with complexities around how do you include things that are local but point to something above? Do we use ID refs? We have to keep it making sense. Should they be part of the same module graph? What does it mean? Because things that are part of the same module graph are singleton, they're one thing, but you could have mutable instances of those. There's a lot of hard problems to work out there. Even when you think this is a problem we're working on, you will find that there are two, three, four interpretations of that same problem that people understand it differently. Specifically on the open stylable, about six years ago, five years ago, something like that, Greg Whitworth, myself, Justin Fagnani and somebody else all worked in a document to describe what we meant by open stylable. There were four different things. Yeah, like you want to be able to do all of them, which means that, well, probably we'll solve these low level things first. Now we have like adopted style sheets. There's people working on CSS modules that you can include. Progress is slow, but steady.
Jay Hoffmann: And so maybe the lesson we can end with is that probably Shadow DOM will never be fully complete as is true of the Web itself. It's been like an extraordinary arc just to get here that spans all of web history. I think we were talking about the second browser ever made. Maybe not the first five minutes, but everything after that all the way to today. It's just one of these really revolutionary concepts, and it's just... Yeah, I think we're going to see plenty more of that.
Brian Kardell: So thanks for joining us.
Eric Meyer: Yeah.
Jay Hoffmann: Yeah, thanks for putting this together. Maybe we'll try this again sometime.
Brian Kardell: Yeah.
Eric Meyer: I think we should.
Jeremy Keith: Yeah, I'm up for it.
Brian Kardell: Let us know what else you're interested in.
Jay Hoffmann: Yeah, we'll get some votes going on a thread or something and figure it all out. Cool.
Brian Kardell
Eric Meyer
Jeremy Keith
Jay Hoffmannn