Webhooks: How a simple idea became a backbone of modern integrations
Igalia's Brian Kardell and Eric Meyer chat with Jeff Lindsay (aka progruim) about the history of, naming and thoughts behind the evangelism of Webhooks.
Brian Kardell: Okay. Hi, I am Brian Kardell. I'm a developer advocate at Igalia.
Eric Meyer: Yeah. And I'm Eric Meyer, also a developer advocate at Igalia. And well, today we're going to be talking about the web like we usually do, but also the hooks that can get into the web. And for that, we have a special guest. Jeff, please introduce yourself.
Jeff Lindsay: And I'm Jeff Lindsay, also known as progrium on the internet. And I guess for this show, it's easy to say. I'm not going to say invented webhooks. I'll say I coined webhooks and evangelized webhooks. Sometimes, I say pioneered webhooks. It's hard to claim an idea like webhooks.
Eric Meyer: Right. But you coined the term is what you mean the-
Jeff Lindsay: Yeah. I mean, there were a bunch of things to it. It was sort of a whole vision, but also the icon that was designed by me and a friend and then just replicated without asking. It just happened, which was the intention.
Brian Kardell: Maybe good analogy is Ethan Marcotte is the one that we typically think of as sort of similar with regard to responsive web design, but he didn't invent all those things, but he described them very well. He drew a box around them. He gave it a name. He gave the seminal introduction to it. So, we kind of credit him as that's the guy that when you talk about responsive web design, he's going to come up. And then Roy Fielding is also a really good example of somebody very similar. He didn't invent hypermedia or anything, but he codified the things around it and defined rest in a way that a lot of people refer to and popularized it as a though.
Jeff Lindsay: Yeah. And I actually am just looking at the Wikipedia about how it's describing his relationship to Rest and also just noticing Rest as an architectural style. I think I describe it as a pattern now, but I think back then, we also said architectural pattern, I guess.
Eric Meyer: Mm-hmm. Okay. So, what is a webhook really? What are you describing? What are the details of what you're describing when you say a webhook?
Brian Kardell: It's what you use to catch a web fish.
Jeff Lindsay: I'm thinking of the fish on the front of the building scalable websites book. Okay. One thing was is I usually would never say webhook that sort of evolved. I would always say webhooks. I also would put a space between obviously people put things together, but the idea was that they're just user defined HTTP callback, which seemed like, I don't know how you feel about that as a description, but one of the things that I felt while trying to talk to people about this was for some reason they wouldn't get that. And I could never tell whether it was, are you not familiar with callbacks or it could just be that it was such a different way to think about HTTP for some people.
Brian Kardell: So, would you say that it's a little bit related to RPC, right? It's a remote procedure call. And would you say that maybe a pingback could be?
Jeff Lindsay: Oh, yeah. Well, let's see. I think pingback comes pretty close. I think I've included that in prior art. And there's also the web, I guess PayPal called it IPN. And there were payment processors using that for that particular use case for a while. I think PayPal was the only one that gave it a name. RPC, yes, because HTTP is basically RPC. It's like you've got the five verbs or whatever. So, you could think of it that way.
Brian Kardell: But I mean there's like XML RPC, which is more specific. I don't know how old you are, so I don't want to age you in my own thing, but when I was getting started, there already had been a pretty long arc of. There used to be RPC and networked RPC even. And then people were like, 'No, no, no. We're going to do this CORBA, common object resource broker.' I don't even remember what the. But basically it became comm and DCOM, distributed comm.
Jeff Lindsay: I love that stuff. I've done heavy research on all of those.
Brian Kardell: Yeah. There was EJBs, enterprise Java beans, all these things were ways for us to have distributed architectures, services, functional programming in a way. I feel like they're all evolutionary trying to learn from, take what they like, what works, leave what they think doesn't work. And I don't know if you think webhooks came a specific time and era where it picked certain ideas that you thought really were helpful and would work or-
Jeff Lindsay: Yeah. I mean, that was the idea. It's sort of like you implement it once to solve a problem and then you realize, well, if everybody did this, then all of these things are possible. And then once you get that idea, it's sort of impossible to let it go. Webhooks was kind of like a drive by thing. I didn't really want to be known as the webhooks guy, but once it started taking off, I'm like, 'All right, I'm done.'
Eric Meyer: So, basically, I mean, it was a thing that was already there, right? It was possible.
Jeff Lindsay: Yeah, a little. Yeah. Oh, yeah. Yeah, yeah. Because some people would be like, 'Oh, it's just reverse HDP.' And I'm like, 'Well, no, because that's in the paradigm of client server that you're not really talking to a client. It's more server to server, but it's also not just server to server because to me, the biggest part of webhooks is the user defined part, which is why I'm not sure ping backs would qualify under my definition, but very similar, obviously.
Brian Kardell: Say more about the user defined part. What do you mean by that?
Jeff Lindsay: Well, today, I mean, I would describe it as an open-ended integration pattern because the idea was. I mean, there's a couple ways to look at it from the perspective of integrations today or at the time what was going on. And I remember a big part of the original motivation was even though 20 years, it's 2026, I registered the domain webhooks.org in 2006, but I think the first blog post and talk on webhooks was in 2007, which was right when Yahoo Pipes had just come out and-
Brian Kardell: Oh, wow. Where was the talk? Do you mind if I ask?
Jeff Lindsay: The first one was at SuperHappyDevHouse, which was an event that we organized. I remember sitting in the front row were the four Twitter engineers at the time. In theory, they said it inspired the fire hose. It would have been webhooks, but then they're like, 'Hmm, at our scale, at our throughput, maybe something else.' The part of the pitch was to avoid polling because we didn't have web socket. We couldn't really do persistent connections, like long polling was even kind of like, because of the infrastructure load balancers and stuff like that, because this was the beginning of cloud era, so we're starting to use AWS and these pre-configured routers and stuff. But it was also at the end of the PHP Apache ring, and that was a big part of the user defined aspect, which is trying to make it easy for. Really the goal was to get user code to run when certain events happen. I mean, they're named after SEM subversion hooks. That was my original use case was exposing-
Brian Kardell: Oh, wow. I didn't know that.
Jeff Lindsay: ... subversion hooks over the web. I was running a service called DejaVu that was kind of like a GitHub, but with SVN. It was based on Track. You guys remember Track?
Eric Meyer: Barely, but yeah.
Jeff Lindsay: Barely. I fell in love with Track and decided to try and run it as a product as opposed to an add-on to hosting account. And so, in that case, you're not going to have SSH access. This is the beginning of SASification of everything. My friend who helped start SuperHappyDevHouse had just made PBwiki and that was like, oh, he just took the simplest open source thing and then turned it into a service. It felt like that was a pattern that some people were doing at the time. We called it peanut butterization named after PBwiki. I was trying to do a startup called DejaVu that was hosted track as a product. How do you expose those hooks? Because those were really powerful. If you know the lean startup thing, I don't know if people still talk about lean startup, which is kind of a play on or extending the ideas of agile to sort of the entrepreneurial four steps to epiphany kind of customer discovery stuff. Eric Ries wrote that book, but he was working at a company called IMVU or MVU, which was kind of just a social chat environment, but they hired a lot of game developers, really talented people, and they had ended up with this really powerful system organizationally that was sort of allowing people to make features. Anybody could make a feature and deploy it and see how it impacts the bottom line. They had the safety to do that because they had good test coverage and a pre-commit SVN hook. So, you couldn't commit code that would break things because it would run all the tests before it even entered the repository. And that was a pretty good, basically addictive system, but a lot of that culture was made possible by this hook because the pre-commit hook is just a script that you'd run, but if it returned a non-zero exit code, it wouldn't allow that commit to go through. And so, that was an important aspect of why it was framed as hooks as opposed to notifications or something like that because that use case was particularly appealing to me as a power user hacker. And my big complaint was web apps are not extensible in the way that non-web apps were. The browser might have extensions and you could get, what was it at the time, Greasemonkey or whatever. But there weren't many applications that were like here's. They would have hard coded integrations. So, part of it was integrations. I really wasn't thinking about it as integrations in general. I don't know that people were at the time because most integrations were kind of like hard coded one-off things. And webhooks, the idea was it's a baseline that you could then allow users to create the integrations in a one-off way. But then if you build on that, for example, GitHub, and GitHub is who I would credit as probably most popularizing webhooks in developer workflows. They had a thing, I don't know if you remember this, I don't think it's still around, GitHub services, which was basically an open source project that would be a bunch of implementations of hooks that would integrate with this service and you could go and add something to it and all it's doing is running that little snippet of code for automatically setting up as the hook. And it even had a little DSL Ruby, so there's a DSL so you can describe what fields are used and it could construct the URL or whatever. But the idea was Webhooks was a power user thing so that I could easily write a PHP script to handle some event, which there are a bunch of use cases that I was excited about and that's what would end up in a lot of my talks. A lot of which were sort of like, 'Hey, and this will enable a world of what we would call no code and services like Zapier.' I wanted all of this stuff to happen and then it did. It just took 10 years. I thought a really good example of the GitHub services thing is how this can be used as infrastructure for a more user-friendly way to let users extend web applications. Today, I would do this with web assemble because the whole point is to run user code near the system that's creating these events.
Brian Kardell: Yeah. I can see why you say architectural pattern almost because it's a little bit like you define these extension points more or less, right? So, I think of a sort of template and command pattern where it's like, well, you have an opportunity to step in here and do something and we'll let you maybe return a value or stop processing or whatever, but you define those things. There's a part of this though that you had talked about that you were like, 'We couldn't use polling.' I think about reverse HTP, but it's not really reverse HTP. I think you could correct me if I'm wrong, but I think that some of this went on to inspire some stuff that ultimately ran at Google and maybe also powered Google Reader. Do you know what I'm talking about?
Jeff Lindsay: Is this the PubSubHubbub?
Brian Kardell: Yes, PubSubHubbub.
Jeff Lindsay: Yeah.
Brian Kardell: Can you explain that a little bit? Because I don't know that that is super incredibly obvious how that works. I don't even think I know how that works because how is that not polling? I think the main service must be polling or crawling or something, but then it raises an event and notifies people or it. Yeah, I don't know. I honestly don't know.
Jeff Lindsay: Yeah. There's so many use cases. One of the use cases was user notifications before we had OS notifications and stuff. People use Scroll and stuff like that. It's like, how do I get that to make a notification if I could get a hook for the event? Or just getting an event stream of whatever that system is modeling is helpful. Once you have enough of those, you have all the state changes, you're able to replicate data in theory if you have enough of these events. So, it was like, 'Oh, I could kind of synchronize two applications in theory if they had webhooks to sort of create effectively a change stream.' But then I was also excited about the extensibility stuff. And I want to ask you about extensible web manifesto at some point, because that was my whole thing. Today we talk about malleability, then I was talking about programmability, which to me was a step towards malleability. So, the extensibility was a big part of it. There were things that you just couldn't do with a rest API or polling, but it was an ergonomics thing because again, it was like the common stack people used was PHP and Apache. So, it was harder to set up a persistent process that would run or set up a cron job or whatever it is. Those were all a little bit more difficult for people that were coming in as just kind of web developers, which was kind of same with JavaScript developers. There's like a funnel. The web is so pervasive that it becomes a funnel where it feels like the majority of developers come in maybe first doing JavaScript and then now there's a whole path to the back end. Back then, it was PHP and Apache, which made it hard to pull or do persistent connections or anything like that. So, the idea of having the other end post potentially with a payload, so you wouldn't have to do anything from that. The idea was some cases, I could make something happen with a two line PHP script.
Brian Kardell: I also was really keen on a lot of these ideas. I'm thinking you're saying 2006. I'm trying to think what was I doing in 2006, but I know somewhere around-ish that time I was working on a project and we were starting to use some subversion. Were they called hooks or actions? I don't remember, but building some pretty complicated, make sure that these happen and make sure that these tests run. And then over time, I got to a point where somewhere around 2010, maybe 2012, I helped build a system that was trying to say that like, 'Okay, look, so you actually have a finite number of pages and the fastest way to serve them is statically.' And so, we could hook these things up to your databases so that any time something changed, we just know how to be ready before they ever ask the question. Time-based caching is weird, right? So, you could build this sort of trigger-based system, this event-based system that just. So, similar to the hooks thing, I mean, I've always thought that there was a lot of really cool ideas that you could do there. We use a number of hooks at Igalia in our GitLab and stuff like that. We have some pretty advanced things that I don't think that we could possibly do without hooks.
Jeff Lindsay: Yeah. I mean, it's an escape hatch. It's like, here's the product, but if it doesn't do exactly what you need or you can think of some way for it to work, hooks kind of allow you to hook in and make that stuff happen. I'm a tinkerer. I would do stuff like use GitHub issues as the database for the thing. I love this idea of reusing existing applications and basically not building stuff. We joked around and called this serverless architecture, and then somehow it became something else, which is also related to webhooks because to get webhooks, I thought to help popularize them, it would help to build infrastructure and tooling. So, I built a bunch of things that were cloned and didn't exist before I made them. There's a simple app engine thing called post bin, like a pace bin, but you get a URL and then whatever's post to it, you see the post data and then we renamed that to request bin and then it's been. For a while, everybody doing webhooks would link to that because it was such a powerful, useful tool. It's useful for general HDP debugging too. So, there's a bunch of those. Another tool that I made was called localtunnel, which you'd run a server locally and then use SSH to create a reverse tunnel and have that to a public server and you'd be able to route traffic to your local web application. And that was useful for debugging webhooks. You're working on them locally. They have to be online to actually get that post. So, I basically wrapped SSH and made a service called localtunnel where you could say localtunnel 8080, and then it gives you a URL that you could share publicly. And then this was cloned. If you look up localtunnel, there's another project called localtunnel that basically erased the history of mine. But then a friend wrote ngrok. We were working at Twilio together, and so that was the blessed one, and that still exists.
Brian Kardell: What is ngrok? I'm not familiar with it.
Jeff Lindsay: Based on what localtunnel did, you can say ngrok 8080, you're running a local web server on 8080 for development purposes or whatever. And then it gives you a URL and you can share this URL. It doesn't matter what network you're on, people with this URL get routed to your local server.
Eric Meyer: Nice.
Jeff Lindsay: So, it's great for debugging, but also demos and quick sharing things. Eventually, ngrok was making a business out of it. And then there was Scriplets, which was a Pastebin-like site where once you hit save, it gives you a URL that just runs the code that you pasted. And so, this was, again, 2006, 2007. The idea was it should be even easier to write handler scripts for webhooks, and then that kind of. The Lambda stuff was basically the same conceptual model for basically the same purpose, like event handling. So, they kind of got to the same place without explicitly being about webhooks. It was more for their internal stuff, but effectively the same thing. Everything I do, I try and do as a public service or get it to run as cheap as possible so that it doesn't cost anything so business doesn't have to be created around it. I mean like webhooks, it's one of those things where surely a lot of people are doing that kind of stuff internally or whatever, but to kind of bring it to the web at large, it's like what business model would ever lead to that? And for a long time it was like I was even trying to think, okay, webhooks are at least somewhat taking off. Is there some way I can do. I didn't want to do a webhook startup, but at the time you also couldn't because there wasn't really a market. I was watching other people do them and fail.
Eric Meyer: So, I mean, what do you feel like caused webhooks to sort of take off and how long did that... You said something like it took 10 years.
Jeff Lindsay: Well, I guess, so part of the idea was if you give a thing a name, then it becomes a thing. Without a name, it doesn't. And that was all I was trying to do was put a name to a thing. And then eventually, people kind of associated me with webhooks and were asking for things like a standard. But even in those, I would always push it off because when you talk about PubSubHubbub, anything more specific than this general pattern is not as general. It would be like PubSubHubbub was made specifically for feeds and you can kind of make that work for non-feed events or whatever and GitHub tried to. And it was an interesting exploration of how do you do webhooks right? How do you verify the sender is who they say they are and all these things that had to kind of be figured out along the way.
Brian Kardell: But PubSubHubbub and the kind of like reverse thing that you're talking about, I think I've heard this referred to as like don't call us, we'll call you kind of thing, right? And I'm curious specifically with PubSubHubbub, is it literally just a enormous bus? What is it? What is it exactly? What was it? I guess it's not really a thing anymore, right?
Jeff Lindsay: Yeah. I was sort of only excited about it in that it was, 'Hey, it's a protocol that's based on webhooks.' And basically I was supporting anything that would use the term webhooks, but it was made by Brad Fitzpatrick who I knew from LiveJournal and another Google colleague of his, Brett Slatkin. And they were just trying to reduce the need for polling, if a Google Reader would have to pull all these feeds. So, the idea was if you get these feed providers to implement PubSubHubbub, then there's a subscription API where you'll get a webhook. I don't know that it had the payload, but then you would know now I can go and check. So, that's one way to do it. You could do big payloads where it's like, here's everything you might need and others where it's like, here's the notification, you know where to get more information. And I think that was one where it would just notify you and then you could go pull and then that would still be a huge reduction of the requests you would have to make, depending on how you do batching and stuff like that.
Eric Meyer: Yeah. So, the PubSubHubbub, you could point your blog at PubSubHub, whatever, and then when you update, it sends a thing to say-
Jeff Lindsay: Yeah. I forget that it was actually made as an intermediary because the way when GitHub implemented it, they just implemented as I'm the pub sub, I'm the hub and the provider. That's what kind of made most sense to me. They had an architecture where you create kind of federated hubs or something, but just the idea of it being a hub is something separate from the publisher and the subscriber did seem a little weird to me because it's like someone has to run that and now you're just adding another system. But yeah, that was the idea. And I guess it was because you're trying to reduce the work of a blog owner basically who would be publishing stuff. So, that made sense. But yeah, I guess it didn't solve a problem for most people. And yet webhooks solved a lot of other different problems as well as that one because it was a simpler, more general. I mean, there's best practices, which most of them are just messaging patterns. It's like any kind of MQs, that whole messaging world, all those patterns. It's all the same thing. So, it's like I've been asked a lot to write something that's like best practices, but it was by then, five, 10 years later, I'm like, 'I don't even care about webhooks anymore. I'm glad they exist, but I've moved on.'
Brian Kardell: Can I ask, what is a question that I've heard asked of a lot of people who coined something or named something or developed a programming language or something, right? What is the sort of most unexpected thing that you've seen webhooks used for that you're like, 'Oh wow, I did not...' Do you have anything like that that you can think of or-
Jeff Lindsay: The thing that really surprised me was more of it being involved in an exploit. I think during the 2020, 2021, there was some scam phishing thing where they stole crypto through Discord and they used the Discord webhook as part of the attack vector. Finally, webhook's in the headlines and it's this, I felt.
Brian Kardell: Hackers be creative.
Jeff Lindsay: Yeah. It's like, 'Should I feel responsible for this? Did I do something wrong?' That's the kind of stuff that happened. Felt like a very simple, high leverage twist to what people were doing on the web.
Brian Kardell: I mean, do you expect it to be as important as it is today? It is so integrated into everything that we do at this point. Did you think that that would be the case? I mean, I guess probably you did because you thought it was a very powerful thing, but-
Jeff Lindsay: It was like a compliment to what normally would be web APIs. I don't know, you could call webhooks a web API, but it just is so kind of different than what you think of when you say API that I usually classify it as something different. And now we're at the point where people don't care. Sometimes people do webhooks and not call them webhooks and that's fine. It was just exciting to see it take off, especially too when we made this little triangle logo and I start seeing it everywhere. But I mean, that was the idea was it should be everywhere. That was the whole point. I wished everything had webhooks that was. I mean, ultimately I wish everything was extensible, but for the time if we got webhooks, then all this stuff would happen. And so, yeah, PubSubHubbub was a protocol. I don't remember too many other. I'm sure there are protocols that involve webhooks somehow, but GitHub sort of... If you think about how part of GitHub's success should probably have to do with how flexible and integrable it was, that you could maybe even think of part of GitHub's success being webhooks or at least that kind of extensibility.
Brian Kardell: I mean, I definitely think so. Internally in Igalia, like I said, we use GitLab and then externally we use GitHub, but externally we also use GitLab, but it's complicated. But yeah, we have so many things like just the integration of a pages concept and the ability to just set up actions and say you do these things and then you do these things and then you can do these things and when those are done, you can do these things. You just move that over to this pages branch-
Jeff Lindsay: Yeah, sort of build this-
Brian Kardell: ... and it's like you're-
Jeff Lindsay: ... machine.
Brian Kardell: Yeah, you're just building this machinery with little to no code.
Jeff Lindsay: Yeah, exactly.
Brian Kardell: Yeah.
Jeff Lindsay: And that was the idea too. I had originally in some of my talks, I would mock these out where it's like maybe there was feeds a tag on a page that says there are webhook events here and then browsers could implement a thing that would be like, 'Oh, and here, write some little code that happens when this event happens.' I was like, 'This would be possible.' Flock, the old good flock from web 2.0 inspired a lot of that because of what they were doing with the browser. There are a lot of things like that where it's like there's this grand vision and then we only get 10% of it, but it's probably the 10% that's good enough.
Eric Meyer: Right. I mean, I remember... So, there was a period where I kept running into this thing of webhooks and there's a webhook and I didn't really understand what a webhook was, right?
Jeff Lindsay: I don't think anybody did.
Eric Meyer: Well, right. But I just kept thinking, what am I hooking? How does the hook... And then eventually when I realized, 'Oh, it's a way to essentially, rather than me having to send a request to get a thing as a developer, as a coder, I don't have to use Curl every five seconds to see if a page has been updated. There's a webhook. I can just wait for it to send me a little notification, PubSubHubbub, whatever.' Or on GitHub in a repository to be able to say, 'Okay, when I update this repository, I want you to ping that thing over there and then it'll do whatever it does.' But either just send in a notification or I don't remember if this was initially the case, but send a whole huge complicated thing for that, whatever that is over there to process. Then yeah, because I got excited because-
Jeff Lindsay: I like this.
Eric Meyer: ... I thought, okay, small pieces loosely joined. This is the loose joining of the pieces. And the pieces don't necessarily have to be small in fairness, but pieces that are loosely joined and not loosely joined in the way that we're used to, which is, all right, I'm going to go check every minute to see if anything's changed. Instead, I have a script that just waits. I set up a little receiver instead of a parabolic dish almost and a little signal gets sent to it and then okay, now I can do a thing. That was much looser joining than I was used to with HTTP since it was being a stateless protocol. It's like, 'Ooh, this is so cool.' I didn't do a ton with it, but it was absolutely one of those, this is awesome that this exists and when I need it, I'm going to be so, so happy that it's there. And that has always been the case. So, was that kind of in your mind, the pieces loosely joined?
Jeff Lindsay: Yeah. Like I said, there was Yahoo Pipes came out around the time. Given just the timing of it all, could have motivated me actually going and starting to share this idea because I remember Tim O'Reilly wrote... because he was John Udell, the original evangelist, Microsoft evangelist, had talked about the internet as a programming platform and Tim O'Reilly, it seemed like he's like, 'And this is it, Yahoo Pipes.' And I was like, 'I mean, yeah, but it's just there's more than feeds on the internet. There's all kinds of other APIs. I would love to program all of that.' And again, I didn't do any of this through a company. I was just a person. Nobody paid for me to go and give these talks. Apparently conferences do pay speakers sometimes.
Eric Meyer: Sometimes. Sometimes.
Jeff Lindsay: Sometimes. Sometimes, yeah. But I would use Pipes, basically the question was what are pipes for the web and is Yahoo Pipes it? And I thought no. And this is a weird metaphor because it's not one-to-one, but I've been obsessed with the composability of Unix Pipes for my entire career. A lot of my current work is related to it, but just infrastructure, right? It's just the ability to say, 'Oh, and this output can go to this input.'
Brian Kardell: Exactly. Yeah.
Jeff Lindsay: But it felt like we didn't have that output side of web apps. And so, even though it's interactive, the input to me was the web API. I talked to it and then the output is it talks to me or representative of an endpoint I own or control. So, to me, it was basically creating the infrastructure for pipe-like composability on the web. And I guess it was I wanted more. I wanted more layers, I think. Those layers kind of exist as products like Zapier. But like I said, it's like, well, it's the smallest, it's good enough. It worked.
Eric Meyer: Yeah, it's the building block.
Brian Kardell: I think when you talk about these timelines and you're saying Yahoo Pipes and things like that, I think that the world generally was on this, 'Ooh, we can do mashups.' All these ideas about what could we do if we had data that we could now smash together in interesting new ways? I just went and looked up even Tim Berners-Lee's first TED Talk was the next web of open linked data. And he was trying to convince people to put their data, make it available as a CSV if you want, just set up some services, publish it, make it available. And he also in that talk showed a bunch of examples of mashups and things that people did. And he's like, 'Look at this. This is what happens when you overlay real estate prices and airplanes and just weird things that somebody was curious about.'
Jeff Lindsay: Housing maps was my favorite go-to, which was Google Maps showing all the Craigslist postings for rent or whatever. The mashup stuff was big. And again, I was like, 'This is great, but a mashup is a composition of two things into a new thing. It doesn't make the two things integrate.' So, I was like, 'That's the difference to me is integration versus a kind of composition.' So, again, it's very similar conceptual... And maybe this is why it was confusing because it's like mashups versus webhooks, what's the difference? Yeah, I think people figured it out, out of necessity, like, 'Oh, I need to make something happen. Oh, what is this? A webhook? I can make code run when something happens?' So, I think most people discover them.
Brian Kardell: Yeah, it's a tool, but I think the point I was making is like you were talking about pipes and Unix pipes and things like that and Yahoo Pipes. I think that's the world of. Sorry, bumped the microphone. That's the world of mashups and things. And I think lots of the world was trying to think about how do we put these things together? And you're coming at it from the other end of it, but you're laying down the foundation of something that lets you put lots of things together. So, if somebody publishes their data, how do you know about it? If somebody updates their data, how do you know about it? I think lots of these things. And also RSS, this is the heyday of RSS, right?
Jeff Lindsay: I guess part of the problem with. No problem, because I love RSS. I loved all of the feeds and everything, but people would talk a lot about data, but there was always non-data use cases. For example, I worked briefly on. I feel like I worked on a lot of stuff and only 1% has made it out in the world, but that's how it works. I was working on a spec for a way to make an HDP request and include a callback URL in the headers so that the response can just post, maybe give you a 20 whatever it is, accept it, and then does this for certain operations that take a long time, which I worked at DigitalOcean on their REST API and originally, it had that built into it so that you could get a... Because VMs took a while to start up in the early days and DigitalOcean was known for like, 'Oh, we start up in less than 60 seconds,' which still sounds like a lot today. But yeah, even 60 seconds or reboot the machine or any of these long operations where you might not be able to hold the connection open long enough or they don't want to hold the... I don't think they ever did that, but that's the kind of... Especially in the broader web ecosystem, people talk a lot about data, but to me, data is one side. It's like code and data. I also think about the code stuff and even though you can describe it as data like, oh, it's the data when the... But it's a mechanical thing. So, it felt like there are a lot of uses for it.
Brian Kardell: Yeah, I mean this reminds me of when I was very young, first started, I went to work for this company and they were a consultancy and somebody contacted us. They had a big, big company, really big company, and they had this database and they would do these reports every month and the queries would take 24 hours to run. They wanted to hook some of these things into web interfaces and you can't keep a connection open for 24 hours. You know what I mean? That's just not how it works. So, you need some way to be like, 'Okay, well, we'll let you know when it's done.' In that case, we definitely didn't have webhooks, but I think we worked out something with DCOM or-
Jeff Lindsay: And it's those use cases that... Because that's why PayPal did it, because you would have the flow where the user goes to PayPal and back, you want them to come back as soon as possible. It's going to go and do fraud detection, there's going to be chargeback stuff, all kinds of things can happen. It takes a while for those things. And so, they wanted a way to let you know of the status update of that. So, this kind of long asynchronous process. I think, like I said, I don't remember what they were, but I remember seeing the providers before, that was the only place others... I mean, other than pingbacks and stuff. I did see in the prior work research, there was an RFC for a notify verb that I think some Microsoft people were pushing, which we never heard about since, but that was an interesting alternative world where I could have said, 'Oh, and we'll use Notify instead of Post,' but it seems easier for people to just use Post than that. I don't know how flexible servers are with unknown verbs. I forget how that works.
Brian Kardell: Yeah, I don't know actually.
Jeff Lindsay: But it's actually, it reminds me of CGI because to me it's a different kind of webhook, but this was the first hook that made the web dynamic because the original spec for HDP was basically... I don't remember what included in 09, but it was mostly a read only kind of... Even the Tim Berners-Lee made the original browser was a read, write authoring tool. Where did I go with that? Oh, the CGI stuff, because Apache, at one point, we all used the same web server. A big part of that was virtual host stuff, being able to multiplex on the host, which of course required a rev of HTTP for that to work, the host header. HTP, one of the few protocols that lets you do that kind of thing. You can't do that with SSH. But yeah, so no web server implemented any kind of post, put, delete, right? They existed, and when people would do APIs, they might get some... Doesn't make sense semantically. That's kind of what REST helped give us a framework to think about that. But REST wouldn't work if there was no mechanism to actually implement those other verbs. So, Rob McCool, I think it was, he was a HTTPd Apache developer. And I love not just then and Web 2.0 where just random people would do a thing and then it's part of the fabric of the internet. Now it's all companies, people at companies, but still feels very like company. But he basically proposed a way... I think what people wanted was a way to search or use the web as an interface to other databases and other systems. And so, that's why it was described as a gateway, originally common gateway protocol, and then you become interface. But all it was, was a format to do a hook. CGI was originally just about running a process when a request comes in, it puts everything you need for the headers and stuff in environment variables. And this is great because you can implement it in any language. The downside is it's starting a process with every request, which we iterated on. We got FastCGI and then later, we got Apache. Most people went towards the Apache modules, Mod PHP and Mod Python, I think, until things started changing with felt like rails and all these other lightweight. But even then it's like the format for... I guess eventually we got to a model where it was just a proxy and that it was easy to write servers like Node.js coming out, you could actually write a web server that would handle post and put. But yeah, the fact that it wasn't possible and then this one little thing, CGI, this little hook, this little escape hatch allowed us to make the web read right.
Eric Meyer: Yeah. So, we should probably wrap up soon, but there was one thing I wanted to go back to, which is you said that all the stuff that you've worked on, 10% or 1% of it's all the light of day. And when it comes to webhooks, is there anything that you really regret didn't develop or that you wish had developed, but it didn't happen?
Jeff Lindsay: Most of it eventually did. I guess I really wanted web apps with plugins and extensions and applications that want to support that, have that ability now, sometimes through webhooks, sometimes through other means. There's a lot of the details, like I mentioned, oh, that there could be a protocol that detects them on a page and then the browser lets you write stuff. There's still not really a great, and probably for good reasons, like Scriplets style, like write some code, hit a button, and now you get a URL, mostly because there's going to be all kinds of abuse. A lot of those things where it's like I want this to exist, but I don't want to start a company or have to deal with that. Sometimes it is necessary, like localtunnel, I actually really wanted to run that as a public service, but eventually the. Okay, here's one thing that's really annoying is because from the beginning there is no way to say stop posting stuff that services like localtunnel or Pastebin or any of these that are made to be used with webhooks will get webhooks that just never stop and that's something that you kind of have to deal with at an infrastructure level, and then there's legitimate abuse and all that stuff is hard to do as a public open source service or something. So, it makes sense that it's companies for some of these.
Eric Meyer: Right. Okay.
Brian Kardell: What would you like to say that sort of wish somebody had asked you over the last 20 years?
Jeff Lindsay: I've gotten so much from learning from history. Every good idea I've ever had has been inspired by something in computer history or something like that, and that it's so much more valuable to look at than staying up-to-date with the latest regurgitation of yesterday.
Brian Kardell: I agree, yeah.
Jeff Lindsay: And maybe it's also like there were people involved in... I feel like today, it's really easy to forget that the web and all this stuff was made by people in a lot of cases without corporate interest or companies involved. And especially, I mean, there were many of these, but the web 2.0 era was one of them and I would love people to go back and look and see how things were back then because it was pretty fun. And we're kind of entering a similar kind of like everybody's building kind of world. And so, it would seem like it would be a good idea to look back at the last time we had that.
Brian Kardell: Yeah. Speaking of people, we were like, 'How do we get introduced to this Jeff guy because we don't know?' But you and Eric have a common acquaintance and through actually something kind of community oriented and fun is-
Eric Meyer: To Kit Hudson.
Brian Kardell: Kit Hudson.
Jeff Lindsay: Yes. Thank you, Kit.
Eric Meyer: Yeah, thanks Kit.
Jeff Lindsay: We go way back and she's done a lot of cool... been involved in a lot of these cool movements and communities.
Brian Kardell: She spoke very highly.
Eric Meyer: She did.
Jeff Lindsay: Probably from the Dev House stuff maybe, which still exists. We had their 70th Dev House last weekend.
Eric Meyer: Oh, cool.
Jeff Lindsay: Most of them took place between 2005 and 2010. We did like 50 in that space and then a couple ever since. But it also spread around the world, it was an inspiration for BarCamp, that spread around a lot. So, I'm involved in a lot of these kind of things that are cool scenes, but it's hard to... Maybe partly because I'm independent, it's kind of hard to... And the fact that every time I make something, I immediately run off and do something else. It's kind of hard to keep track of I'm getting older. The other part of trying to get webhook talk about, because I don't care about webhooks, but I do care about the story and the vision that was originally there and how... I don't know, sometimes it takes... It's like I saw the next 10 years at least, but it took 10 years for it to happen. So, there's sort of a humbling patience about this kind of stuff.
Brian Kardell: Yeah. Cool. Super thanks for coming on and giving us your time. How can people follow you or give you money or get in touch with you?
Jeff Lindsay: Yeah, because you can give me money. I do have a GitHub sponsors. That's mostly how I've been operating. I do contract stuff as well if people are interested in collaborating a bunch of cool new stuff, including Wanix. If you go to Wanix.dev, it's basically Unix in the browser, but based on plan nine architecture and it'll kind of live beyond the web, but I think you guys will find it a little mind melty, which is my favorite thing to do is melt minds. But program on any Twitter program on YouTube. I have a YouTube channel pretty much anywhere you can look up Program.
Brian Kardell: Bluesky or Mastodon?
Jeff Lindsay: Yep. Yep. Or you can just go to program.xyz and links to all of that.
Brian Kardell
Eric Meyer
Jeff Lindsay