In this episode of the IoT For All Podcast, Ryan Chacon is joined by François Baldassari, Founder and CEO of Memfault. They discuss various topics, including device management, embedded software, and best practices for IoT developers. The podcast is kicked off with an introduction of Memfault, who they work with, and a high-level overview of how the product works. They then talk about the industry as a whole and how it’s been evolving. They conclude with advice for IoT developers and how to overcome challenges François has encountered in the space.

François Baldassari is the Founder and CEO of Memfault, the first observability platform for IoT devices. An embedded software engineer by trade, his powerful passion for tooling and automation in software engineering drove him to start Memfault. Previous to Memfault, he ran the firmware team at Oculus and built the operating system at Pebble. François has a BSc in Electrical Engineering from Brown.

Interested in connecting with François? Reach out to him on Linkedin!

About Memfault

Memfault is the first cloud-based observability platform purpose-built for IoT devices that brings the flexibility, speed, and innovation of software development to hardware development. With Memfault, developers can remotely debug issues, deploy OTA firmware updates, and continuously monitor fleets of connected devices at scale. By pairing crash-reporting and metrics collection tools with firmware update and configuration management capabilities, developers can proactively monitor and manage device operations, from development to feature updates in production, from one cohesive interface.

Key Questions and Topics from this Episode:

(01:24) Introduction to François

(03:20) Introduction to Memfault

(06:11) What is Memfault’s product made up of

(07:02) Memfault’s partnerships

(08:25) How has the industry evolved

(10:48) Challenges IoT developers encounter

(13:50) Best practices of device development and production

(17:45) Challenges seen in the industry

Transcript:

- [Voice Over] You are listening to the IoT For All Media Network.

- [Ryan] Hello everyone and welcome to another episode of the IoT For All Podcast. I'm your host, Ryan Chacon and on today's episode, we have François Baldassari, the CEO and Co-Founder of Memfault. He, the company is focused on being the first cloud based observability platform, purpose-built for IoT devices that brings flexibility, speed and innovation of software development to the hardware side of development. So we talk a lot about Embedded Software and how it's changed over the last decade. The challenges IoT device developers have had in monitoring, updating and securing devices. We also talk about some of the experience that they're able to share around the building cutting edge IoT devices, for things like Oculus and Pebble, and what they took away from it, what advice they kind of share with you, as well as best practices for IoT device management during the development and the production phases. So, a lot of good information here. I think you'll tap on a ton of value in this conversation but before we get into this episode, if any of you out there are looking to enter the fast growing and profitable IoT market, but don't know where to start, check out our sponsor, Leverege. Leverege's IoT solutions development platform provides everything you need to create Turnkey IoT products that you can white label and resell under your own brand. To learn more, go to iotchangeseverything.com that's iotchangeseverything.com and without further ado, please enjoy this episode of the IoT For All Podcast. Welcome François to the IoT For All Podcast. Thanks for being here this week.

- [François] Thanks Ryan, it's a pleasure to be here. Very excited for the conversation ahead.

- [Ryan] Same here. Let's kick this off by having you give a quick introduction about yourself to our audience.

- [François] Yeah, absolutely. So my name is François and I'm an Embedded Software Engineer. I've been building devices that you've heard of like the Pebble smartwatch, the Oculus Quest Virtual Reality headsets. And I started my career building much less exciting devices when I built servers at Sun Microsystems. But I think more pertinent to what we're talking about today, I'm the Founder and CEO of a company called Memfault. We have been thinking really hard about Embedded Software and IoT, and how to make the lives of engineers who are building the future easier and better.

- [Ryan] Fantastic. So I guess I have to ask you before we dive in further about your, what you did with Oculus.

- [François] Yeah, so I joined Oculus originally as just another Embedded Software engineer. It was, shortly after the Facebook acquisition. And at that time, you know, Oculus was growing like crazy. I think that I joined, there were maybe 500 people by the time I left, I think there were 5,000 people. So you can imagine it was a lot going on. But the really exciting thing I got to do is work on Oculus Quest and the transition from this PC VR, where you plugged in your VR headset into a PC. And then into a standalone VR and these kind of all in one devices. If you've never tried Oculus Quest, it's an amazing experience.

- [Ryan] I have one, that's why I had to ask.

- [François] Yeah, I was lucky enough that I ended up leading the Embedded Software team, so I got to see not just, you know, the code itself, but also think about how do we build these devices and how do we make these teams more efficient?

- [Ryan] Fantastic. So, let's, let's talk a little bit more about Memfault and the company, on your, from your side of things. Talk about a little bit high level, kind of what you all do, role you all play in IoT. And I'm also curious to just hear about the opportunity you saw to kind of build this, and kind of bring this, you know to be a company.

- [François] Yeah, I'm going to, answer your questions a little bit backwards, and I'll start with the how, because I think it explains the what we do. You know in 2014, I was an engineer at a company called Pebble that made smart watches. So, you know, devices on your wrists that get your notifications and have sensors, all kinds of things. And at Pebble, you know, although we were firmware engineers building firmware, the, you know, the way that we all build firmware, you know, we were struggling with a lot of things that actually are traditionally software problems, because our watch was so complicated, it had a multi-threaded operating system, it had third party applications, you could install apps on it. It had a UI, you know, because it was so complex, we had a lot of software running on it and so we updated it every couple of weeks, introduced some new features, introduced some new bugs and inevitably we'd spend our lives just tracking down problems like the Bluetooth not connecting, or the watch, you know, running out of battery, things like that, that would take, you know, all of my time. So people would just mail those watches back to us and we cracked them open. It required a hammer and a heat gun, 'cause they were glued together. It was a really arduous process. And then we'd connect our debugger and we'd eventually figure out what might have happened to the device. And that just felt crazy to me. It was such an intense kind of effort to just understand what had happened to our customers. And you know, next to me was an iOS engineer, a guy who was working on a mobile app. And when I told him, how do you do this? He showed me, he was using a piece of software at the time called Hoki app. And whenever someone had a problem, anywhere in the world, he'd get a report automatically with a back trace, with a memory, with everything he needed the logs to fix the problem. And I thought, I want that. And so from there on out, I went on a mission to build tools, to give me the same kind of observability, the same visibility and diagnostics that software engineers have had, but from our hardware devices. And that's what we're building at Memfault. We're giving you the power of observability, the power of remote diagnostics in the context of an IoT product, and I'm very happy to say we built it at Pebble for ourselves. It made our lives so much easier. And now that we've been selling it and building it to companies like Skip and Whoop, SunPower, Verkada and many others, we've gone to see that same transformation operates. And if you want, I can give you more details about how this works, but hopefully that gives you a good idea of what it is.

- [Ryan] Yeah if you could kind of break that down at a high-level to talk about how it works, that would be fantastic.

- [François] So at a very high-level, our product is made up of two pieces. First, we have the SDK, so small piece of C software, very very small, it's about two kilobytes of code space and one kilobyte of RAM. It works on pretty much any connected device. And we implement that SDK into your device through a firmware updates, takes about two weeks of work. And that little SDK collects signals from your device. It collects kind of faults being reported. It collects battery level, it collects Wi-Fi statistics, whatever else, and uploads them to the clouds where we analyze all these data and give your engineer like day by day insights on what problems your customers are experiencing, and how you can fix them.

- [Ryan] Fantastic. That's awesome. So I did wanna, you mentioned some interesting brands that you work with. You mentioned you work with Whoop. Is that, what are you able to kind of tell me on that front?

- [François] Yeah, so I can tell you a, a little bit, of course, I wanna be mindful of their confidential information, but long story short, we've been very excited to work with the folks at Whoop, who like we did at Pebble make a wearable device that you wear on your wrists. And of course like we experienced at Pebble, once they reach scale, I mean, Whoop makes a lot of devices, they're everywhere these days. Of course, they start having in the margin, you know, a few problems that keep creeping up. And when you have a hundred devices, you know, a problem that happens once every 10,000 hours, you never see it. But when you have a million devices, a problem that happens once every 10,000 hours, all of a sudden is happening all the time and is hammering your support team and is driving your RMAs. So with Whoop, our focus has been on giving them the visibility they need to nail down those issues, fix them instead of spending hours of customer support time, tracking them down one customer at a time, and drive that RMA numbers down, drive the number of customer requests down and make their product better and better. And we're so excited. We've, they've actually been increasing their involvement with us every year. So they're a great customer.

- [Ryan] So let me ask you, as we kind of talk about the Embedded Software side of things, how, from your perspective, being in this space for such a long time, how has it kind of changed over the years? You know, maybe over the last decade or so, kind of where, where, how, how have we evolved to kind of where we are now?

- [François] I've seen a crazy transformation. I, you know, if I think about what we did when I started my career, not so long ago, right? I started my career in 2007, or started working with embedded devices in 2007. At that time, what you could get for $1, as a Mac controller, it had what, like two kilobytes of RAM. It had maybe a couple kilobytes of Flash. And so at the time you really didn't write software. What you did is you wrote a little control loop, you might have, you know, a little sensor acquisition thing, and you'd logged out to serial, but you certainly weren't building complex data structures, today, if you buy the top of the line, STM 32, you get two megabytes of RAM. You know, you get like, you get basically a computer for a couple bucks and super low-power consumption. And so people are starting to build real complex software on that. I've heard of people doing, you know, Machine Learning at the edge, right? It's like companies like Edge Impulse, where I've seen people doing computer vision at the Edge. I remember chatting with folks at a company called VergeSense, who does, you know, did a bunch of machine vision, on their embedded devices. This is, you know, hundreds of thousands, sometimes millions of lines of codes. And so necessarily it's changing the way that we build these devices and it's changing the way that we kind of envision the long term value that they provide. Increasingly I think you'll notice two interesting trends. First IoT companies, have way more software engineers than hardware engineers, in 2022. Used to be the opposite. They used to have way more like electrical, mechanical engineers, but that has switched. The other thing is, they're adopting software business models. So they're no longer just selling devices for a fee, what they're usually doing is they're selling a service. They're selling consumables, they're monetizing over time because that's how software is sold. And so our industry is actually turning into much more, much more of a software industry than a hardware industry. And that transformation has been dramatic and it's changed the way we work.

- [Ryan] Absolutely. That's great, thanks for kind of breaking that down for us. I did wanna kind of, so as we've been talking about more physical devices in this conversation, what are some of the challenges IoT device developers have had in, you know, monitoring, updating, securing devices as compared to, you know, the software development side of things?

- [François] Yeah, I, you know, this is a thing that led me to create this company. In the software world, when you need to figure out how to monitor something, update it, kind of maintain it. You don't have to reinvent the wheel, right. There is eight different companies who will sell you off the shelf software. And there's a very well established way to solve those problems. In the IoT worlds, you know, there's, really not enough off the shelf. There's not enough like software that you can buy to solve those problems and address those challenges. And so what are people doing is they're building it. And so we're looking at companies who are startups, they have 30 employees and here they are building an entire OTA systems from scratch. They're building entire log connection, collection systems from scratch. They're just building an entire set of infrastructure, not because they want to, not because they wish they had, they were excited about hiring infrastructure engineers to build this, but because it's the only way, and in 2022, you need a way to update your devices. Like it's no longer good enough to just put a device out there and stop worrying about it. And so, and you need a way to monitor your devices. So these companies make that, realize that, and they start investing heavily in infrastructure building it. But every dollar and every head count they spend building an infrastructure is dollars and head counts, that don't go towards building better products. And so, so I think that this has been a major challenge, and again, and again, we run into companies who tell us, we've just spent $2,000,000 building custom infrastructure. We wish we'd found you two years ago. Right?

- [Ryan] Absolutely. That's it's always been an interesting conversation when we kind of talk about the two different sides of this, right, the software side and the device, the hardware side of things. So it's interesting to kind of get your perspective and, really tie back into the company you built is actively solving a problem, that is very, very apparent.

- [François] Yeah, absolutely. And I think that, you talk about the software side versus the hardware side. And it's funny because in our heads, those are such different dichotomies. We really think about hardware and software as different disciplines. Where I'm seeing is increasingly, they're doing really, really similar work, but they're doing it very, very differently. And so over time, I believe that we will unify engineering practices and we will unify the way we work. And that we'll look back at this time where we thought of hardware and software as complete distinct things, as a time where we hadn't figured it out yet.

- [Ryan] Sure, and in IoT, I mean, it kind of, they, I mean, they, they have to work together. I mean that's how we're successful in the deployments or we're not, if they're not thought about in the same light. So that's a very good point. I did wanna also kind of ask you to share some thoughts on, if we're, if you know, we have device developers out there, companies that are working in device management, what are some of the things you've noticed from like, that are best practices when it comes to the development and production of devices themselves and kind of integrating that in on the embedded side, in the, on the software side and so forth.

- [François] Yeah, I think there, there's three things that companies who build hardware maybe need to spend a little bit, or increasingly will find themselves thinking about. The first one is security. I think that historically we've gotten away with, I think maybe fewer investments in security, in the hardware space than the software industry has had to reckon with. And increasingly it's coming back to bite us. Right? We see, you know, IP cameras being repurposed to create botnets. We see people's devices being remotely disabled or being used to spy on folks. You know, for example, I own a baby monitor and I'm very, very mindful that that not be used to watch my baby from the internet. Right? those are real concerns. That concerns. And so I think that, you know early in the development cycles, companies need to build a security strategy and it can be as simple as starting to secure your OT updates, right? Like it's, you know, cryptographically sign your firmware to make sure nobody else is loading firmware on your devices but you. That's number one. Number two is, you know, employ cryptography to secure the link between your device in the clouds, whether you're just doing, you know TLS or even MTLS just thinking about this up upfront and employing standards where possible, it will save you a ton of time. And more, more importantly, it will save you from that terrible, terrible phone call. You might receive two years from now, about the fact that your device will compromised. So that's, I think that's top of mind, the other one is of course sustaining. I think as hardware engineers, we get so focused on shipping, you know, it's you go through EVT, DVT, PVT you work so hard. You, you know, go to the manufacturing lines to debug test fixtures, and then the product is out and you feel like you're done. And the reality is that that today that's no longer true. You're not done when the device leaves the factory, because these devices are connected to the clouds. They provide ongoing value, they're they potentially need to evolve as the customer need evolves. And so really we need to start thinking about not just how do we, the next two years look like to get this device out the door, but what does what a five year strategy to make sure those devices provide maximum value to our customers. So between that and security, I think those are the two bigger blind spots I see.

- [Ryan] Yeah. Security's an interesting one because it's the conversations I've had so far have really been around when does security need to be thought about in order to be brought into an IoT solution successfully. And it's really like never can be too early to be thinking about it, but oftentimes security is an afterthought for a lot of people and it's comes back to bite them later on down the road. So I think the fact that you kind of emphasize how important that is crucial to, for people to really understand.

- [François] Yeah. Security should be part of your electrical design. Right? And so that means it's the very, very beginning. There's nothing worse than thinking you're almost ready to ship your product. And then you do your security review and you realize that you need to add a security, a secure element to the bottom right? Now you've gotta spin the board. You've gotta redo all those tests. You've gotta, you know, change your calibration and manufacturing processes. And you just delayed your product by three months. There has to be a secure security review at the design stage. And I think that that's a mindset change.

- [Ryan] Yep. Absolutely. I, definitely, definitely agree. One of the last questions I wanted to ask you kinda, before we start to wrap up here is around challenges you've seen in the space. And this can be from any perspective, but one area I'm really curious to hear your thoughts on is around the view device developers, the device creators have when it comes to having certain constraints around what's physically possible versus what the customer, is looking for and expecting or asking of them to build. Because there obviously are different constraints that, you know, the buyer may not necessarily understand or want to understand, but they're realistic. So how do you kind of handle, or how have you seen those types of conversations go? How do you see that playing out in the market? And what kind of advice do you have for both sides of you thinking about that a little bit differently?

- [François] I think everyone who's ever built a product has run into this frustration, which is, you know, you put your product out there, you get it in the hands of your customers. And all of a sudden they start saying like, why didn't you just dot dot dots? And they're telling you, why didn't you just add this feature? Why didn't you just do this or that? And, that's, I think it's because we make look so easy. You know, companies, great IoT companies make it look like it's nothing to build and ship and scale a fleet of IoT devices, but it's actually very, very difficult. And throughout the project, we make really hard trade offs. And oftentimes it's not physics that limits us, you know, some companies run into physics, right. If you're making, trying to build AR headsets, there are major physics challenges to solve. But more often than not it's costs. Right. And so what I find is oftentimes people say like, well, why can't I install, you know, three more things on this device, or why can't I save like 10 hours of video rather than five hours video? And the reality is like adding a bigger flash chip would add 10 bucks to the price of the device and people aren't willing to pay those 10 bucks. And so I think the first thing is talking about those trade offs a little bit more and explaining that engineering is about finding the best bang for the buck, which customers really relate to. The other thing I found is that it's oftentimes when customers ask about something very, very specific, usually behind it, there's a deeper needs that is unaddressed. And if you can get to that deeper need, you can oftentimes learn something important for your business. So for example, a customer might say "Oh, I you know, I get a 10 second snippet from the camera when a cat walks by and I wish this was a 20 second snippet." And, what the customer really, the reason they're complaining is because it takes what they want to know is they wanna be able to identify who it is that walk by. And for some reason, the camera activates too quickly. And it's not enough time to see who walked by. And I think if you get to that basic needs, I wish I could identify better who walked by. There might be software solutions. You might be able to kind of change when you start recording, or you might be able to change kind of how you explain things or what you display in your application. And so, you know, always getting to the bottom of customer needs, can avoid you some serious trouble and avoid you from building things that can be really expensive when a cheap fix might have addressed their underlying needs. So I hope that helps. It's a complicated question of course.

- [Ryan] Yeah. I think a big piece too, is just on the understanding where both sides are coming from, and also the educational component, the more understanding somebody has coming into that equation from, especially from the customer side on what's possible, and the limitations and the potential for what they're trying to, have built the better. If they come in with real unrealistic expectations, you kind of run into these problems. And it's a very frustrating process when you're on the technical side for sure. So that makes a lot of sense. And I appreciate you kind of shedding some light on that. 'Cause it is something that I don't think people really understand how much of a, a road block that can cause at times even, when you kind of get into those kinds of conversations.

- [François] One thing I'd like to add on this topic is that oftentimes customers, what they say is different from what they really want. And so I always push people to look at the data, right? Like make sure you collect the right information so that you can actually make some data driven decisions about what your customers are actually doing with your product. And that's something of course that we helped with there are other ways to do it, but the most important thing I'll say is, get your hands on the data and see what you can actually read from that.

- [Ryan] Absolutely. That's great advice. This conversation has been very enlightening. Thank you so much for taking the time to do this. I did wanna ask. So for our audience out there, who's wants to learn more kind of what Memfault's doing, kind of you know, what you have going on, follow up anything we talked about here today, what's the best way they can reach out.

- [François] Yeah. So the best way to reach out is to find us on our website at Memfault.com. That's the name of the company.com. You can also find us on Twitter. The handle is Memfault. Hopefully easy enough to remember. Last but not least, we're always excited to hear from people. So don't hesitate to shoot us an email at [email protected].

- [Ryan] Fantastic. And anything kind of on the horizon that we should be able look out for and paying attention to coming outta of the company soon.

- [François] What I'm really, really excited about is the idea of fleet sampling, which is, you know, imagine you have 10 million devices out there. And you wanna know what the top problems are with them. You know, one thing you could do is instrument all 10 million devices, but that's gonna cost you a lot of money cuz it's a lot of data. But what we find is that if you instrument 10% of those devices, or even 5% of those devices, you can get 90% of insights. So this is something we're working on, selecting the right sample of devices to give you those insights and tell you what your top problems are for 10 times cheaper.

- [Ryan] Wow. Fantastic. Well, we'll be in a lookout for that and we'll make sure we link up the Twitter handle, the company website, all the different information about the organization and all the, the push that we do when we promote this, all the descriptions and so forth. So I really employ kind of our company our audience out there to, to reach out and learn more about what you're doing. So I think what you have going on is fantastic. So thanks again for your time. Really appreciate it.

- [François] Thank you, Ryan. And thank you for your extremely thoughtful question. I love the conversation.

- [Ryan] Absolutely. All right, everyone. Thanks again for watching that episode of the IoT For All Podcast. If you enjoyed the episode, please click the thumbs up button, subscribe to our channel and be sure to hit the Bell notification. So you get the latest episodes as soon as it become available. Other than that, thanks again for watching. And we'll see you next time.