As design teams shrink and hardware engineers increasingly face challenges with embedded software, the industry confronts a widening skills gap, resistance to modernization, and pressure to deliver faster. But the solutions to these challenges exist.
Jacob Beningo, founder of the Embedded Software Academy, has worked on over 100 projects across 20 countries. He spoke with EE World about practical strategies for upskilling teams and modernizing embedded development practices, including abstracting your architectures from hardware dependencies, using AI as a force multiplier and not a replacement, and building continuous education into your team’s DNA.
———
EE World: Design teams are getting smaller, and we’re hearing from more hardware engineers who are suddenly responsible for embedded software tasks for which they weren’t trained. What are you seeing in terms of this skills gap?
Beningo: “There are certainly folks who are electrical engineers or they’re mechanical engineers, and they’re trying to, you know, they get thrown into embedded at work, and now they need to understand, okay, how do I do this embedded thing, right? A lot of folks who get into embedded come from electronics backgrounds. My degrees weren’t in embedded systems; they were in electronics engineering. And then, you know, the hardware ties very closely to software, right? You like hardware, and you kind of have an interest in software. It kind of bridges that gap area.”
The challenge is accessibility. “It’s so much easier to go and build an app for your phone or to do web-based applications, versus I need to go and get hardware to do something. I think they’re just less exposed to it. And so it’s the question, how do we go about showing them that there’s more to the, you know, the software and in our electronics than just our cellular devices and mobile and web apps.”
EE World: In your experience working with embedded teams across multiple industries, what technical practices are holding development back? Where are teams stuck using approaches that made sense 10 years ago but now create problems?
Beningo: “A big one that I focused on with teams is actually their software architectures. As electronics professionals, particularly in traditional firmware and embedded systems, we tend to think about designing from the hardware, right? And that’s great, because hardware is important. However, when you do that, people tend to tightly couple their code to a specific piece of hardware, and we can’t easily change it if a customer has a new request. What happens is that you end up with a bunch of spaghetti code.
“The alternative is actually to look at things from a top-down perspective. I often say the modern philosophy is that there is no hardware, only data, and so we allow data to dictate the design. So, you look at hardware and say, ‘Well, it’s a push button.’ Well, from a push button, people will be like, Oh, well, what’s the debounce rate, and what GPIO pin is it connected to? We say, well, we don’t care about those details. We’re going to abstract them, so we don’t care about those details, and we just need to know: is the button pressed or not?
“If you follow the data flow of everything input and output from your system and how it’s transformed, where it’s stored, it actually creates a nice architecture that decouples from the hardware and is very scalable and flexible. When a customer comes and wants some new feature, you’re not going, oh man, now I gotta go and adjust these 15 modules.”
EE World: That’s a compelling case for modern architecture. However, change is challenging, especially when teams are under pressure to deliver. Are engineers receptive to this approach, or do you face resistance?
Beningo: “It’s very dependent on the team. Some of them are very open to thinking that way. Others, very much, are kicking, screaming, resisting. Whenever you tell an embedded person that you ignore the hardware, they generally get all up in arms, because we love the hardware. But COVID taught us, right? If you’re tightly coupled to it and then suddenly your parts are not available, now you’ve got to rewrite everything, instead of just saying, Well, let me just rewrite this little piece of the software, and the rest of my stuff is okay.”
EE World: AI is everywhere in tech conversations right now, but embedded developers seem more cautious than other software communities. Where do you see AI actually providing value in embedded development versus just hype?
Beningo: “There’s actually a lot of interesting things to do there. I personally am slightly leery, just because I focus a lot on quality systems, just because I worked in defense and automotive systems, where when you write software, it’s gotta work, right? I’m very leery about letting agents just go off and write code, because a lot of them have been trained on open source software, which is a very poor representation of how you should actually write software.
“But that being said, I think that there are a lot of interesting techniques that you can use for letting AI do code reviews, or assisting code reviews, letting AI generate small snippets of code, working with AI as an intern to accelerate boilerplate types of code, and helping you architect pieces of software. It’s super cool from a process standpoint.”
EE World: You mentioned using machine learning on-chip. Can you give a concrete example of where this approach outperformed traditional algorithm development?
Beningo: “There’s also a lot that can be done with machine learning on-chip. I had a customer I was working with, and they had a sensor to be able to do gestures, like to raise the volume, you move your hand in front of the sensor for this kid’s device. We collected a whole bunch of sensor data and, instead of trying to come up with our own algorithm, we just fed it in and trained a machine learning model that we then generated C code for and ran it on-chip, and it worked much better than the algorithm we originally tried to design.
“I actually designed one originally. It took me something like six, maybe eight weeks, to get something that was working 90% of the time or above. Then, two years later, some of these machine learning tools came out. Within several days, I had the equivalent of a machine learning-trained model that worked as good or better than the one I had spent six weeks fine-tuning. Machine learning can help with essentially managing sensor inputs that normally would be really hard to train models for.”
EE World: You’ve talked about technical solutions—architecture and AI. But what about the human side? How should teams approach ongoing skill development when everyone’s already stretched thin with project deadlines?
Beningo: “Just doing simple code reviews, people skip them a lot of times, or they’ll do them asynchronously. Just by doing code review, doing code reviews is the best way to get rid of bugs in your code, in addition to sharing knowledge about what it is you’re actually building. You often end up with senior engineers and junior engineers in a room reviewing code. It’s actually a great learning opportunity, as well as the lowest-hanging fruit to get bugs out of your system.
“Generally, I will encourage people to identify those times in your calendar where this is your education. It should be, in my opinion, on a weekly basis. They should at least be spending an hour, if not more. I personally spent 30 minutes first thing in the morning every single day looking through articles and projects.”
EE World: Let’s talk about the practical implementation side. You mentioned build systems earlier—many teams are still using hand-written makefiles. What’s the path forward there?
Beningo: “Transitioning into using CMake-based systems, using containers, adjusting their build system in such a way that they can easily pass the container to a colleague, so that they don’t have to set up the tools themselves, that everybody’s working on the same tool chains with the exact same tool sets. I’ve seen issues where teams have chased themselves for a day or two trying to figure out why there is this bug, or the comment ‘it works on my machine?’ And it’s all because somebody upgraded a compiler, and there’s a bug in that compiler.
“By doing that, you can pass around the containers to each other, but more importantly, it enables CI/CD. Those same containers can be used for continuous integration, which can help build out how they test their systems.”
About the author
Jacob Beningo is an embedded software consultant, founder of Beningo Embedded Group, creator of the Embedded Software Academy, and co-founder of the Embedded Online Conference. He helps developers and teams maintain and modernize their embedded software skills and processes so they can deliver better firmware faster. He holds two bachelor’s degrees from Central Michigan University in Electronics Engineering, Physics, and Mathematics. He also has a Master’s in Space Systems Engineering from the University of Michigan. Beningo’s Embedded Software Academy offers online courses covering CI/CD, agile practices, simulation, software architecture, and modern embedded development techniques. Learn more at www.beningo.com
Leave a Reply