General tips on debugging embedded hardware

When you first start out after graduating from college with a B.S. in electrical engineering, you have a bright, shiny new degree that might not be exactly what you expected. You had to take a series of fundamental courses in math and science for the first year or two. The third year is spent learning everything from good, old-fashioned AC/DC power, the basics of electrical behavior, and you might take courses on basic microelectronics by the third year. Somewhere in there you also have computer programming courses, digital logic, and math that starts to make sense in the context of solving engineering problems. In short, it’s difficult to specialize in anything with a bachelor’s degree. Lab work and senior projects are not always in alignment with what find yourself doing on your first job.

This article is for the newly minted electrical engineers who find themselves debugging embedded hardware. From a very high level, some general tips are as follows:

1. Don’t assume you are supposed to know everything. Cut yourself some slack; you got an engineering degree that taught you how to be an engineer, not how to solve every engineering problem that is thrown at you. You learned how to learn like an engineer. Engineers never stop learning. Patiently figuring out a problem means methodically going through all the possibilities.
2. Don’t try to debug the whole system or PCB at once. Start off with the most obvious options first. Does it have power? Are the cables good? It’s ok to briefly check the whole system by answering some obvious questions first, but if you don’t solve the problem fairly quickly, you need to start narrowing down where it’s originating from by eliminating known good areas, sub-systems, or circuits and even sub-circuits. If the problem is deep or very confusing, start by narrowing down the problem to one section at a time to see where the problem might be coming from, and use a system of elimination to verify each section as good.
3. Use the scientific method: don’t randomly adjust things and expect that a solution will evolve. Even if you get lucky, you still need to know how what you did solved what specific problem. Symptoms can vary based on a number of conditions such as operating temperature, how long the circuit is in operation (burn-in), and the test procedures that you choose will all play into what you see happening with the hardware. Patience is key in methodically going through the steps to verify each sub-system or sub-circuit so that you can isolate only one problem (or design flaw if this is initial testing) in a sub-circuit. Narrowing the problem(s) down to one small area, fixing it, then moving out from there to a successively bigger picture is going to save you a lot of time in the end. Trying to fix more than one problem at a time can lead to confusing results and a lot of frustration. If software is possibly involved, decouple the source of the software from the hardware and eliminate the possibility of issues originating with the hardware first. Use bench equipment to deliver known good signals coming from another area if you need to. You can use bench equipment to deliver known bad signals that you can use to try and improve results or to observe how known good circuits upstream of the signal chain are handling signal issues. Whatever you decide to do, make sure you investigate related issues while you are in that particular test set-up. Again, patience and curiosity will win the day.
4. Take copious notes. Information is your friend later on when you find a pattern and need to verify what happened in an earlier test set-up. Some of the activity that you see might seem quite normal at the time, but turn out to be key to your investigation. What behavior were you expecting? You have to first understand what is supposed to be happening in a circuit before you can clear the circuit as “known good.” What is it actually delivering? If the behavior is very confusing, identify key specifications that would need to be met as you move through each component in the signal chain. Is your original assumption based on solid, realistic expectations? Test various expected operating conditions that have changed to possibly cause the problem. It is also possible that you are operating a component or circuit in conditions outside one of the specifications for that component. In rare instances, it could be that the component does not operate as advertised under a few specific conditions. For example, an op amp could be oscillating under specific load and signaling conditions only and could be a design flaw in the component that was not caught by the manufacturer. Don’t be afraid to try another component or a similar component from another manufacturer. If you have narrowed the problem down to one component, don’t be afraid to assume that the manufacturer is at fault, and contact the product support line if you cannot find a quick substitution for the component, with a different part number if necessary.

The importance of good notes, even in seemingly innocuous testing areas, can save you from having to re-investigate an issue. For instance, if you test a circuit operating at several voltage levels that have not seemed to cause the problem, take notes anyway on your test set up, the voltages you tested at in the circuit, and what you saw (even if they were “normal” results.) Even if you test thoroughly, take full notes so that you will at least know what areas you need to further test in that area if finer detail is required later on. That way, you will not have to guess at the test conditions that you used earlier and you can quickly set up the test bed again, but this time adding a little twist that you discovered might make a difference. As you gain experience, this is most true on areas that you are not 100% absolutely expert. However, even the most experienced engineer is always learning new oddities in areas that he/she thought they already knew.

Successful engineers are patient, methodical, curious, and stubborn; which means they are willing to doggedly continue to find possible solutions. In fact, most design engineers would prefer to have an infinite amount of time to create, experiment, tweak and test a design, but only hobbyists have this luxury. Therefore, the real challenge for engineers of all stripes is in delivering a good working design on time and under budget. With time on the job and experience in methodically looking for, finding, and solving problems with your circuit design, you will find yourself building a library of experience that is invaluable.