Brian Kardell: Okay, hi, I'm Brian Kardell, I am a Developer Advocate at Igalia.
Eric Meyer: And I'm Eric Meyer, also Developer Advocate at Igalia. And today we're going to be talking about multimedia, and for that we have a special guest. So special guest, please introduce yourself.
Thibault Saunier: Hi, I'm Thibault Saunier. I am a multimedia engineer. I work on, well, anything multimedia related around video editing, video conferencing, any video processing basically. I am a gstreamer core developer and maintainer. So I work mainly with gstreamer, on gstreamer, making sure that upstream works and that the project works properly.
Brian Kardell: So let me ask a question that probably a lot of our listeners would have. I definitely had it when I came to Igalia. Gstreamer, what is that?
Thibault Saunier: Yeah. So basically, gstreamer is a multimedia framework. What it does is it allows people to retrieve streams from anywhere, process them the way they want, and then broadcast them or play or record them locally, things like that. It allows you to, basically you have what we call elements, which is like low level bricks that you can put together. So some elements will do video decoding, for example. Some elements will retrieve multimedia stream from an HTTP server, things like that. And then you can put them together to build whatever you want with them. And then you can have elements that we call sync that will just stream the result of all the processing that you did to some other places. I am mostly working on video editing based on that world framework. It's just a brick on top of all the basic bricks that we have in gstreamer. So it allows you to build any videoing features that you might have based on gstreamer. And the nice thing is that we have lot of components for many different use cases. For example, you will have components to build a video conferencing system. For example, you will have WebRTC integrated in gstreamer, so you have an implementation of WebRTC, which allows live communication between people. And WebRTC is a web standard, so basically we implement many web standards, and this is one for live video conferencing. And for example, you can reuse that specific element to build anything you want so you can do all the processing you want and then just stream live through WebRTC or you can... That's a very powerful, very modular.
Eric Meyer: Okay. So WebRTC you said, that's Web Real-Time Communications, right?
Thibault Saunier: Yep.
Eric Meyer: Is what that stands for?
Thibault Saunier: Yeah, that's correct.
Eric Meyer: Right. So that's the standard of doing stuff in real time.
Thibault Saunier: Yeah. Communicating between people in a web browser usually, but you still have things happening server side. So you have, in the browser, you have implementations of WebRTC so that you can build video conferencing systems in the web browser.
Eric Meyer: Okay. And so when you say video editing, I think of video editing as I'm clicking and moving stuff around on my screen saying, 'This is a clip and I want to take this clip and I want to move it over here and I want to trim this part off,' and all that sort of thing. How does that happen on the server side though?
Thibault Saunier: For example, we are building a complete video link application that runs server side, and then you have, it's a web interface so it's a web application. It's Sequence.film, the name of the product that they're building on top of gstreamer and the editing services. And there you have a full video editing application in your web process. So basically you have your timeline, you have everything. And what happens is that you have your clips, as you said. You move your clips around, you trim things, you put your effects, you do your collaborating and things like that. And what you see is actually happening server side, and we just send a live stream with WebRTC. The video that you see in your web browser is actually just a WebRTC stream. So live stream that is streamed out of the timeline that is running in the cloud. And then you see the video in the web browser and nothing happens locally. All the decoding, all the video processing, nothing happens locally. You just receive a stream from the server, and that uses WebRTC as I was discussing before. That's something that is allowed by the very modular framework that gstreamer is. You just have a WebRTC component that allows you to just have your videoing timeline that trends server side, and then we just stream that through WebRTC to the web browser. So for that particular use case, the focus is full-fledged video editing, but we also have other approaches that target different audience. For example, we've been working with Tella, which builds a video application that allows you to produce nice looking content without really knowing what video adding is. It's a much simpler user interface and allows you to create marketing content or courses, things like that in a very simple way. And then it also runs all server side the video processing and all that will happen server side and very efficiently. So the user, the same way as for Sequence, they don't need four full computers to do that. For Tella, they basically need a webcam and being able to have decent video internet connection.
Brian Kardell: I mean, yeah, those both look like really cool projects.
Thibault Saunier: Yeah.
Brian Kardell: Eric, I think the way that I think about this is like if I was implementing a front end, like those pictures that you might see, the stills on a timeline, they're just stills, right? And then like at some point I would want to select a time chunk and say, 'Delete this.' But I would, on my front end, I would think I would translate that into like a command that I would send to the server that says, 'I intend to delete this.' You would do that on the server, done, and then the front end would update that it was done.
Thibault Saunier: Yeah. But at the same time, you will see in the viewer so in the video that you have, because always in video editing application, you have the timelines and then you have the video that is the result of the timeline. So what happens is that if you cut some parts and you were, your play head was at that position, it will just update the video. But what happens is that you send the comment as you said, remove that, that it will happen server side and then you will have the actual video resulting from that edit coming in directly in the process.
Brian Kardell: Right, yeah.
Eric Meyer: Well, I'm curious, so because of my background and just the things that I've seen over the years, my assumption would have been that basically all of that was being done in the browser. What's the advantage to doing the editing on the server as opposed to locally?
Thibault Saunier: Well, in that case, the very interesting part is that you can have collaborative editing basically.
Eric Meyer: Oh.
Thibault Saunier: So you have several people working on the same project and then you have all your media assets are actually stored in the cloud so you don't have to have huge amount of memory of this on your computer. You don't need big computers with any... Well, you can just go and edit with a tablet while the processing should be very expensive in that case if you're doing collaborating and complex effects and things like that, but it all happens in the cloud so you just need some computer, a good internet connection, a decent internet connection, and that's it. You can edit with-
Brian Kardell: And you don't have to worry about, 'Oh, I accidentally dropped my laptop and the hard drive is destroyed.'
Thibault Saunier: Yeah.
Brian Kardell: Hopefully.
Eric Meyer: We're back to thin computing, woo-hoo.
Brian Kardell: Yeah. I mean, okay. So gstreamer is a framework and this is one of the things that you can do with it, but it's like other things that are used for video editing, it can be used by a lot of things like FFmpeg is a thing that I've used a lot. It's used by a lot of things. I think it's also even used by gstreamer, is that right?
Thibault Saunier: Yeah, that's right. So basically gstreamer aims are allowing people to use different technology that implements different things. And FFmpeg is one of those things that implements very well decoding and things like that. While it can use that, but it reuses many libraries and perhaps many libraries and allows developers to use the same common API to work with all those libraries in a coherent and simple way, basically.
Brian Kardell: So it is used on server, it's used in web browsers as well, right?
Thibault Saunier: Yeah. Well, for example, we have a port of WebKit, WPE WebKit, that uses gstreamer for all the multimedia web standard implementations, so it's at the very bit of all processing there. The nice things with using gstreamer is that on any platforms you can just use the same API and then you will be using the video decoder for that very specific embedded platform. But at the webinar level, you don't need to care about all those decoders and all that because gstreamer does just hide it from yo. So you can just use the same API web.
Brian Kardell: I think there's work to do this in Servo too, right?
Thibault Saunier: Yeah. In Servo, the base multimedia implementation is also based on gstreamer. It's not very advanced yet, but yeah.
Brian Kardell: But that seems big given the thing that you said, because WPE webkit at least currently is Linux. It's for embedded Linux. We are working on Android port, which then would be able to take advantage of the thing that you're talking about where it's a different operating system and you can take advantage of the stuff for that. But with Servo, it's available on five or six platforms already so having the same API be able to use on all those seems like a really good advantage.
Thibault Saunier: Yeah, that's for sure. And that's like with the streamer, you have all those platforms also covered works. For example, in my case, I work with my clients often on macOS and then you have it working on Linux, obviously that is the main place where we use it in the cloud, and then it has support for an Android, iOS, everything. So yeah, you implement the back-ends for several ones and then it should work everywhere while you will have issues specific to platforms sometimes.
Brian Kardell: With WPE, and embedded systems, and the fact that it works on embedded systems and Linux, and I imagine it's used in a lot of... I mean, I don't imagine, I know that it's used in a lot of products, but just give you an opportunity to maybe talk about some of those use cases and things that it is used in so that people who don't do this kind of thing get a sense of what kind of things it's used in.
Thibault Saunier: Yeah. Well, on setup boxes, in many of them, for sure. And then you have in TVs also like LG and Samsung have their own multimedia backend based on gstreamer. Samsung in particular used to work there and they had, everything for Tizen is gstreamer based so they have their own... Well, four kind of, not really, but they have their own version of gstreamer with a few tweaks on top, but mostly they have their own plugins because one thing that is interesting with gstreamer is also that it's plugins based. So it's very easy to extend the streamer for your use case, just implementing your plugin. So you have two types of API. We could say one for plugin implementation and one for usage of the API. And so yeah, for embedded platforms in general, it's used heavily in many different places. Then you have many apps on Android and iOS that are gstreamer. That's used gstreamer in some places, sometimes it uses both the native APIs and then gstreamer for the things, things like that. Then in conferencing system, so that's more server side again, video conferencing. And for example, systems that allows to have people coming from different platforms like Teams and Google Talk and different things that there are systems that are just streamer based that allows people to just communicate between all those platforms. They don't even notice that they're talking with different platforms. And then for AI system, also it's quite heavily used. For example, NVIDIA itself actually offers, well, mixed media framework for AI, basically very focused on AI and it's actually all gstreamer based. They have the bulk of gstreamer and that's what they propose their user to use. I think that's interesting because there are tons of systems that actually use gstreamer through their own thing. And upstream gstreamer is not really happy with that because it's not properly done in many places, I think. And we are basically working on making sure that we have all the right tools upstream for AI. So for video analysis and then all the metadata and anything you can imagine with AI on video, right? More about analyzing videos than generation, because generation is now, you don't really need a streamer for that, I think. Yeah. And yeah, another place where it's very heavily deployed is for IP cameras. So you have manufacturers that build a camera with all the multimedia processing happening inside the camera that is customer based, and then it sends the streams to the surveillance platform that they have and they have big systems for all the video surveillance that you can get inside cities and inside wherever you want. And those are also the receiving sites also streamer based. For example, Access is a big gstreamer user for 20 years or something. So they build all those on the streamer.
Eric Meyer: Wow. Ring doorbells, for example, the doorbells with cameras built in, things like that?
Thibault Saunier: I haven't seen any people... Well, no one called us to build that, but I would be surprised if there is none. It's most likely some people use gstreamer for those almost for sure. I've never had a client doing that, but.
Eric Meyer: Okay.
Brian Kardell: I know a lot of people use things like WPE for digital signage as a really popular use case or something that I don't know what you call it, but like... I don't even know if you have them in Europe, but here in the US, if you go to the gas pump, they have a screen that is interactive, and a lot of times they're like, 'Oh, another opportunity to show you ads.' And they play video, so I would imagine that's another use case?
Thibault Saunier: That's for sure. I've worked with many people doing that actually, for digital signage in general. You also have the case where you have big screens with 10 screens that have to be exactly synchronized and things like that so they've built product on top of that. Also, one thing that is interesting I've worked on in the past is little, how call that? To listen to music and you have several speakers in different rooms and things like that. So you use your streamers to just synchronize them together so that you move around and it's all playing nicely and synchronized.
Brian Kardell: Oh, that's cool. Like Sonos had that where it was IP coordinated, so yeah.
Thibault Saunier: So yeah, those kind of products, I've worked on those and gstreamer is used in many cases. In the framework, you have a very well-designed way of synchronizing things together, I can guess, and synchronizing through the network and things like that, so you have those components to load that and make it simple, well, simpler and simpler. It used to be a bit more complex, but these days it's getting really simple to just get things working and synchronized between whoever you want.
Eric Meyer: Yeah. So that ability to do things synchronized, displays and audio output speakers, et cetera. So gstreamer is not really just video. It sounds like you can use it for more than just sending video back and forth.
Thibault Saunier: Yeah, right. Well, it's just data processing in the end. So one interesting example is the legal project, which is a project that discovered the gravitational waves, they were actually using the streamer to detect the waves. To do the data processing, and they had huge pipelines with hundred and thousands of elements, so they are the only elements that were doing some data processing inside, and then that's the component that allowed them to do the analysis of all the... And removing the noise and all that. I don't know exactly details, but there was a conference before they discovered the waves that presented how they used the streamer to actually do the data processing. And it's interesting because you can basically process whatever you want, but most people just use it for audio video, right? But you can process whatever you want so it's very flexible in that manner.
Eric Meyer: Yeah. Any stream of data, basically you're saying you could do stuff with, do something to, or act upon or whatever. So you could use it for, as they did, for signals analysis, in their case they're looking for gravitational wave signals. You could use it on a sonar system like an underwater sonar or radar or something like that to try to pull signals out of that stream of data. Oh, that's interesting because I mean, when you think about it, well, when I think about it, because I don't usually think about it, as I'm thinking about it now, video is, yeah, it's a stream of data. And if you have good ways of handling video or audio streams and applying filters and doing other, transforming them, then yeah, fascinating.
Brian Kardell: I mean, those are at least audio still, but I imagine based on what I'm hearing, you could do something with a real time stock data or something like that that's completely unrelated to sound or video in any way, but you could still do some kind of... I don't know what kind of processing you would do on it, but it sounds like you could do that.
Eric Meyer: Potentially, yeah. Or seismograph data. That would be interesting. If you can use gstreamer to detect gravitational waves in laser interferometry data, which is what LIGO or LIGO uses, you could probably detect earthquake precursors if you knew what to look for out of a whole bunch of seismographs, right? All of the seismograph data coming in at once and then using, I guess you could use gstreamer to combine it all and then analyze it. Ah, that's fascinating.
Brian Kardell: So what we're saying is all of the scientific community, call us.
Eric Meyer: Right.
Brian Kardell: We can help.
Eric Meyer: Gstreamer all the data. James Webb be another example of stuff that could be analyzed, the deep sky surveys that are happening, et cetera.
Thibault Saunier: I'm living in Chile and I talked a bit with some people that work on the huge telescope. I can't remember the name right now.
Eric Meyer: I mean, they have several now.
Thibault Saunier: No, the biggest one that is Atacama, near San Pedro. And over there, basically it was explaining me that they were building a system to do exactly what gstreamer does in the end. And we said, okay, it's too late, because they've been working on that for a few years, but it was very similar. What they wanted is actually what gstreamer could have provided from the start.
Eric Meyer: Fascinating. If only we had talked to them sooner.
Thibault Saunier: Yeah.
Eric Meyer: Maybe that's the very large, whatever it's called. I mean, there's one that's I think literally called the very large telescope.
Brian Kardell: Oh, maybe we could analyze some certain data with this.
Eric Meyer: Yeah. Oh yeah, absolutely. The large hadron collider data that I'm sure you could use gstreamer to do stuff with that based on what I've just learned.
Thibault Saunier: ALMA it's called, the one I was talking about.
Brian Kardell: Okay. So Igalia Chats, we're largely focused on the web. Eric and I's background is very important part of what we do at Igalia. But I think one of the things that's really interesting to me is that there are a few projects that are super complicated and somehow underpin a lot of the things that the web platform does make possible. So like gstreamer, or Skia is another one that's just like, these are the phenomenally complicated projects. I just want to maybe talk about, what is the size of this project? Do you know what I mean? How many lines of code, how many people work on it? How big is the community? Just to give our listeners a sense of what that's like.
Thibault Saunier: Yeah. So I think it's three million line of codes more or less. And we have around 200 contributors every year, so people provided some patches. For co-contributors, people that actually commit every month, let's say, I think we are 20, 25 people I think. And they're from many companies in the end. I don't know how many companies, but if you look at the gstreamer conference that happens every year, we're around 150, 200 people every time, and people coming from 30 companies, 40 companies. There are several companies like us that do a consultancy around gstreamer, and then many companies that are just building products on top. And I think we have 5,000 commit a year or something like that. So it's quite big, not as big as web engine, right? But still.
Brian Kardell: No, but I'm saying the web engine relies on also some of these really big projects to make really complicated things possible, right? I mean, to do all this complicated stuff, to show video, to make embedded systems that show video and stuff like that, gstreamer is really important part of that. If you had to implement all that stuff yourself or find different ways to do it would be more difficult and wouldn't work on so many platforms. And yeah, I just think that some of these big sort of projects that empower many platforms that have to do with font rendering and text shaping. There's a few big complicated projects that wind up being shared that are themselves big, complicated open source projects that deserve and provide a lot of funding into the ecosystem that then at the top, web browsers are one of the things that take advantage of them and you can build great things out of, but they're not the only things.
Eric Meyer: Yeah. Besides, you mentioned Sequence as a thing that we've worked on.
Brian Kardell: Can you actually talk about that a little bit? What exactly is it?
Thibault Saunier: Well, it's a full video application. So if you look at it's the same as other Premier or something like that, but the goal is to have it work exclusively in the cloud and then you have a web front end. So you just connect to the webpage and then you access everything you need to edit your videos. Live with interactive... Well, everything you need. Basically, I like to see it as the Google Docs of video editing. You can see it like that. So collaborative videoing in the browser. Yeah.
Eric Meyer: Okay. What are some other things that we've been doing with gstreamer?
Thibault Saunier: Well, many of them... One thing that is also interesting is we were building a 3D modeling application that also happens, so it's a web app also, you have a web front end and then everything happens server side. So that it's also a thing clients, as you said before, where you just get into the webpage and you interact with the page, but it actually sends the comments directly when you click and all that, it sends the comments to a gstreamer pipelines. And the source of the gstreamer pipelines is actually where the 3D modeling is happening. So in that case, it was a Q-DAS source and they were just rendering their 3D models live in the source and then we just streamed that through WebRTC to the web browser. So it's remote computing in some way, but very, very specific with very optimized path.
Eric Meyer: Okay. Again, let me see if I've got this right. So I connect to the server that has a 3D model of a table and I want to rotate the table in my view. So I do stuff on the screen, all of my input data is sent to the server and it does the actual change rotation and the video of that is streamed back to me. Yeah.
Thibault Saunier: And what you're interacting in the browser, what you're interacting is actually a video. So it's a video stream you interact on.
Eric Meyer: Right. So where I clicked and where I dragged and all that, et cetera in order to spin the view. I don't have a model on my computer and I'm not spinning the model on my computer. My browser is not doing the math to rotate the view. It's just sending my inputs to the server. The server's doing that stuff and I'm getting a stream back of the video of the server doing those things. And also if I click on a tool to add a shape and I start adding the shape, the same thing's happening. None of that's happening locally. It's all streaming between the server and me. That's, wow.
Thibault Saunier: It's interesting. And that's the kind of things that gstreamer allows you to do quite easily.
Brian Kardell: So I don't know the answer to this question, but when we were at WebKit contributors meeting, the people from, I want to say NVIDIA, I could be wrong. They come every year and they're really big on cloud gaming. And I know Sony also has some interests in this area and a lot of them have now things with lightweight glasses that even some are tethered and some are not. But if you put on those glasses, they are literally just streaming the video. There's no big processing on the glasses on those, it's just literally streaming to you. So it needs to be lightweight efficient and that's all it does, it doesn't do more. And I'm curious, do you know if anybody is using this for cloud gaming type things?
Thibault Saunier: Yeah. Well, it's the same thing as what I was saying about.
Brian Kardell: That's why I mentioned it.
Thibault Saunier: Yeah. I don't know any company that uses gstreamer. Maybe they do, we just don't know. For example, NVIDIA, when they do things with gstreamer, they don't tell us, we don't even know. In the end, sometimes we figure out at some point, but I don't know any company that does that. I know we've been working server side to have WPE WebKits work properly with game web services. But server side, I don't know. I can't remember anymore. I'm quite sure there are some, but.
Brian Kardell: You say sometimes we find out and there are things that we work on that we don't talk about, but almost all the things, if you can interact with it, that you can find where there's the open source licenses and you can see a lot of our work in those things. So sometimes we discover you're on a train or something and you find a thing in there and you click, check, 'Hey, is there any of our software on here?'
Thibault Saunier: Well, that's also use case where we have gstreamer deployed in many cases the plain screens who have been working on that, and many of them I think use gstreamer.
Brian Kardell: Yeah. If they're frustrating, they're not gstreamer.
Thibault Saunier: No, no. Those are not.
Eric Meyer: There we go. Didn't we also do some work about Vulkan Video and gstreamer?
Thibault Saunier: Yeah. Well, we have been working on a bit on the standards, but mostly on implementing Vulkan Video support in gstreamer. One of the first actual users of the new Vulkan Video API, which is providing the same API for all the platforms to do hardware, video decoding and coding. Well, we already have a standard that supports most codex and that work on most platforms, but Mac and well, Apple-based products don't have support for it. But for the others, eventually we'll have drivers for Linux and Linux Windows and Android basically, hopefully.
Eric Meyer: Okay.
Thibault Saunier: And for embedded platforms also, it's interesting because right now for hardware codex, on a big platforms, it's a bit of a mess because they all have their own APIs and things like that. It's getting better because we have V4L too, but with Vulkan, it will make everything simpler, I think.
Brian Kardell: I wonder if we can talk about this because like I say, Eric and I talk about the web a lot, and sometimes we talk about how Igalia ourselves invest in things and why we do that. And we talk sometimes about work that we are kind of allowed to talk about like, 'Oh, Bloomberg Tech funded our work on CSS grid and here's why they did it.' And IO, the ad blocker company funded our work for CSS has pseudo class and here's why they did it, and there are all these success stories in there, but I think there's a lot of stuff that Igalia does that's not the stuff that we talk about, and it might be interesting to talk about, how do we end up working on that? How do we end up working on Sequence, that film? How do we end up, why do companies approach us? Do companies approach us? How do we end up working on these things?
Thibault Saunier: Yeah. Well, for video editing, I've been working on that on my free time for a long time in gstreamer. And then basically we built, well, back then for Nokia, we built adjustment services, which is the component that exposes the required APIs for video editing. And then basically we got contacted by, for example, Sequence or Tella, which is another company that does video editing with the streamer. And then they ask us to... Well, sometimes they ask us to help them with the API, but usually it's not the case anymore. We have good enough documentation and all that. They ask us to, well, push forward some features that are missing or optimize that. That's often what they require. They need the thing to be as optimized as possible for their specific use case, so that's something that happens a lot. And while we still have many features for full post-production scenarios, we still have features that are missing or still a bit buggy because many features that we have in GS, for example, have been developed by Google, some of code student or people like that. So happens that we need to finalize the work that has been done by students. Many times we have features that are very specific to the use case that they need, and then they just ask us to implement that.
Brian Kardell: Can you give us an example of that?
Thibault Saunier: For example, we have nested timeline support in GS that allows you to have timelines, clips that are actually a timeline beneath so that you can, some people work on a scene and then you have the full movie where you have some scenes that are... So you put the clip that is just the scene and then you have the other scenes, et cetera. And as they want to be collaborative, they need... If someone somewhere is working on a specific scene and the other one is working on the full movie let's say, they want the scene that is being edited by someone else, the changes to that scene to be a visible right away when you are on the full movie.
Brian Kardell: Ooh.
Thibault Saunier: See what I mean?
Brian Kardell: Yeah.
Thibault Saunier: Okay. So that's something that they wanted and we just implemented that. So you have live updates of sub timelines, which is not, the way we built the feature didn't imply that because didn't think it was really useful. But for that case where you have collaborative editing, it's really nice to have everything happen together and update it live and all that.
Eric Meyer: Yeah. I could see very similar for audio production since we were talking about that. Producing music, for example, if you had an album as a timeline and then each song is a timeline. And if it's a progressive rock band, then each chapter of the song is its own timeline, all nested inside of each other, and then people can be, 'Yeah, that's interesting.'
Brian Kardell: That is actually interesting. I wonder if, to produce this show, we use Audacity and I would actually much prefer a web interface, especially one of those collaboratives so we don't have to... I wonder if we could try using Sequence or something like that to do that. Is it a use case that would work for that kind of thing or?
Thibault Saunier: I guess so, yeah, probably. Also, one thing that is interesting, like I was saying, we also work for Tella, which is another web app, but they are more focused on, right now they're more focused on presentation, let's say. So you want to market a product or something like that, so you have your screen and then you are recording yourself presenting the product or it's not necessarily a product, it can be a course or whatever. You just are presenting something, and then they're able to edit that in a few seconds basically. So first, they have AI that generates a potential timeline that you can then edit, but it tries to build the thing, and it has nice feature, a nice transition, and then it will try to zoom where it needs to zoom and then zoom out when it's not needed and put the faces in the right place and move the... When they see that what is important is the presentation, then it will focus on the presentation, then go back to show the person speaking and things like that, tries to automate that. So you have the final render. And the thing that we do is basically, well, all the drawing and the rendering. And then we are able to render the full presentation in a few seconds because we render each chunk in parallel on different servers, so it's very fast, it's nice. And one of the next feature that we are going to work on while there are, and we are next here is basically building a product where it will be able to do the same thing, but have several people either talk together or present something. So in the end, it will be able to prepare a timeline with the edit for you and doing the transition when someone is speaking. Well, in the case, if it's only audio, it's a bit less interesting, I guess, but still that's also the use case we have that is quite interesting, I think.
Brian Kardell: Okay.
Thibault Saunier: And for editing that kind of conversation that we have right now, I think that will be the ideal tool once it does everything. The thing is that they already have all the infrastructure through the simple presentation thing, what they need is just to be able to have several people discussing when they are doing the presentation.
Eric Meyer: Yeah, that's really interesting. So we've talked about cool stuff that we've done in gstreamer. Are there any things that you would particularly love to work on that's gstreamer related, but we haven't really had a chance to do that?
Thibault Saunier: Well, I'm very happy to be able to work on video editing in particular and all the feature that but there is one thing that is... Well, as I discussed a bit about that before, but we really need to, I think it's important these days to have a good story, a good API, good foundation for AI in particular in gstreamer where we have almost all the pieces to be able to do whatever you will need with AI for AI video analysis and all that. And we have been working on bringing the required APIs on gstreamer. Well, that will work a bit more on that and understanding more everything that is doable thanks to that because many people just use web services to do their things, but I'm sure that we could give a better user experience for developers that require, well, video analysis in many different ways at the same time, things like that, because gstreamer also allows you to use different inference engine and different things and bring all that together, so that's it.
Eric Meyer: Okay. So when you say AI, AI can mean a lot of things these days.
Thibault Saunier: Well, in that case, it's really about analysis, with analysis. It's not really generation, not generative AI, but rather.
Eric Meyer: Okay. So it's more integration such that it's easy to tie in an AI transcriber or something that analyzes the video in some way and then maybe produces a transcription, or I think you were saying suggests zooms and pans and things like that.
Thibault Saunier: Right now, there are web services that allows that, but for example, for Tela, if we could have just gstreamer do everything, because already it would make a lot of sense in that case, but we don't really have that yet. Even though there are models that are available for these kind of things, but yeah, I think it can be useful.
Eric Meyer: Okay. Or I guess hooking up a neural network analysis to Seismograph and James Webb Telescope data streams or whatever, stuff out in Atacama. Yeah, okay, so that makes sense. So those links are tenuous? Are they still being written into the specification or there's a specification that hasn't been implemented? Where does that stand?
Thibault Saunier: Well, we have new APIs upstream that people need to use more. So yeah, I think it's all well specified. The APIs is not fully stable yet, but you can describe any kind of metadata because that's mostly what it is about and having the good relationship between metas and things like that. But then you need to be able to run that on any model and then you need specific things to decrypt the data that the model generates and things like that so it has to... Well, we have to build upon what we have right now. For example, I think for video editing also, you can build nice things for... Well, instead of spending so much time looking at the, I mean, watching the footage that you have, and then put yourself, the metas, but what is happening and when and all that, that should be automated, I think. That's the kind of things that would make sense. And that's also one thing that having video editing application in the cloud that allows you to just run that automatically and gather the data. And so it's just part of, because I don't think you can run, most of the time you cannot run the inference model on the machine you are editing on, right? So you will have to send the footages and then have the analysis happen there and then, yeah.
Eric Meyer: Okay. That makes sense.
Brian Kardell: Eric was asking what's something that you would like to work on. I guess I would say we have a lot of people who listen to the show for various reasons. They work for various organizations, various companies. I would ask if there's something that you would say to those organizations about... Give your pitch about what's something that maybe gstreamer could do for them that they might want to come talk to us about.
Thibault Saunier: Well, basically anything multi-major related, if you have anything to do with audio, video, or live or not, and post-processing and all that, that's for sure we can help. Yeah. And anything in the process for Google PE and all that, we know everything in there, so that's what we can help most. I'm convinced that gstreamers will be used much more for post-production use cases. Many times, people just use FFmpeg with their own big scripts and all that, and we offer much better interfaces to do video editing than what FFmpeg can do.
Brian Kardell: Excellent. Thanks. I enjoyed talking to you and learning more about what gstreamer does. I like doing these things that are a little bit outside our wheelhouse. Hopefully our listeners also like the occasional change up where we get to learn about something completely different and completely new that we do. Yeah, thanks again for joining us.
Brian Kardell
Eric Meyer
Thibault Saunier