Break Points

Faster, Better Code

There exists a silver bullet that can drastically improve the rate at which you develop code while also reducing the number of delivered bugs.

Though this bit of magic can reduce debugging time by factor of 10 or more, despite the fact that it’s a technique well known since 1976, and even though neither tools nor expensive new resources are needed, few embedded folks use it.

Formal code inspections are probably the most important tool you can use to get your code out faster, with fewer bugs. The inspection plays on the well-known fact that “two heads are better than one.” The goal is to identify and remove bugs before testing the code.

Effectiveness
Those who are aware of the method often reject it because of the assumed “hassle factor.” Usually, few developers are aware of the benefits that have been so carefully quantified over time. Let’s look at some of the data.

The very best of inspection practices yield stunning results. For example, IBM manages to remove 82% of all defects before testing even starts!

One study showed that, as a rule of thumb, each defect identified during inspection saves around nine hours of time downstream.

AT&T found that inspections led to a 14% increase in productivity and tenfold increase in quality.

Hewlett-Packard found that 80% of the errors detected during inspections were unlikely to be caught by testing.

HP, Shell Research, Bell Northern, and AT&T all found inspections to be 20 to 30 times more efficient than testing in detecting errors. IBM found inspections gave a 23% increase in productivity and a 38% reduction in bugs detected after unit test.

So, while the inspection may cost you 20% extra time during coding, debug times can shrink by an order of magnitude or more. The reduced number of bugs in the final product means you’ll spend less time in the mind-numbing weariness of maintenance, as well.

The inspection team
The best inspections come about from properly organized teams. Keep management off the team! Experience indicates that when a manager is involved, usually only the most superficial bugs are caught, because no one wishes to show the author to be the cause of major program defects.

The inspection team consists of four formal roles: moderator, reader, recorder, and author.

The moderator, who must be very technically competent, leads the inspection process. He or she paces the meeting, coaches other team members, deals with scheduling a meeting place and disseminating materials before the meeting, and follows up on rework (if any).

The reader takes the team through the code by paraphrasing its operation. Never let the author take this role, as he or she may read what was intended instead of what was actually implemented.

A recorder notes each error on a standard form. This frees the other team members to focus on thinking deeply about the code.

The author’s role is to understand the errors found and to illuminate unclear areas. As code inspections are never confrontational, the author should never have to defend the code.

An additional and optional role is that of trainee. No one seems to have a clear idea of how to create embedded developers. One technique is to include new folks (only one or two per team) in the code inspection process. The trainee(s) then gets a deep look inside of the company’s code to develop an understanding of how the code operates.

Reducing the team size by sharing roles is tempting. Bear in mind that Bull HN found that four-person inspection teams are twice as efficient and twice as effective as three-person teams. A code inspection with three people (perhaps using the author as the recorder) surely beats none at all, but do try to fill each role separately.

The process
The inspection process consists of several steps; all are required for optimal results. The steps are:

Planning. When the code compiles cleanly (no errors or warning messages), and after it passes through Lint (if used), the author submits listings to the moderator, who forms an inspection team. The moderator distributes listings to each team member, as well as other related documents such as design requirements and documentation. The bulk of the planning process is done by the moderator, who can use e-mail to coordinate with team members. An effective moderator respects the time constraints of his or her colleagues and avoids interrupting them.

Overview. This optional step is a meeting for cases in which the inspection team members are not familiar with the development project. The author provides enough background to team members to facilitate their understanding of the code.

Preparation. Inspectors individually examine the code and related materials. They use a checklist to ensure they check all potential problem areas. Each inspector marks up his or her copy of the code listing with suspected problem areas.

Inspection meeting. The entire team meets to review the code. The moderator runs the meeting tightly. The only subject for discussion is the code under review; any other subject is simply not appropriate and not allowed.

The person designated as reader presents the code by paraphrasing the meaning of small sections of code in a context higher than that of the code itself. In other words, the reader translates short code snippets from computer-lingo to English to ensure the code’s implementation has the correct meaning.

The reader continuously decides how many lines of code to paraphrase, picking a number that allows reasonable extraction of meaning. Two to three lines at a time is typical. The reader paraphrases every decision point, every branch case, and so on. One study concluded that only 50% of the code gets executed during typical tests, so be sure the inspection looks at everything.

Use a checklist to be sure you’re looking at all important items. For firmware, a typical checklist covers the following areas:

• Does the code meet the firmware standards?

• How large and complex are functions?

• What’s the clarity of expression—if it’s not clear, it’s junk

• Does the code promote encapsulation?

• Are all external functions declared with complete prototypes?

• Are all data types correct?

• Are variables correctly initialized at the start of the function?

• Are variables correctly initialized at the start of each loop?

• Will the code work?

• Are comments in clear and correct English?

• Are all error conditions caught (such as return codes from malloc())?

• Do all switch statements include a default case (if only a subset of the possible conditions used)?

• Is the syntax correct? Does it properly use ==, =, &&, &, and so on?

• In ISRs, is the code generally reentrant?

• Is performance-limited code implemented efficiently?

Record all errors and classify them as major or minor. A major bug is one that if not removed could result in a problem that the customer will see. Minor bugs are those that include spelling errors, non-compliance with the firmware standards, and poor workmanship that doesn’t lead to a major error.

Why the classification? Because when the pressure is on, when the deadline looms near, management will demand that you drop inspections, as they don’t seem like “real work.” A list of classified bugs gives you the ammunition needed to make it clear that dropping inspections will yield more errors and slower delivery.

The code itself is the only thing under review; the author may not be criticized. One effective way of diffusing the tension in starting up new inspection processes (before the team members are truly comfortable with it) is to have the author supply a pizza for the meeting, thus becoming the good guy.

At this meeting no attempt is made to rework the code or to come up with alternative approaches. Simply find errors and log them; let the author deal with implementing solutions. The moderator must keep the meeting fast-paced and efficient. In fact, a reasonable review rate is between 150 and 200 non-blank lines per hour.

Note that comment lines require as much review as code lines. Misspellings, lousy grammar, and poor communication of ideas are as deadly in comments as outright bugs in code. Firmware must do two things to be acceptable: it must work, and it must communicate its meaning to a future version of yourself—and to others. The comments are a critical part of this and deserve as much attention as the code itself.

It’s worthwhile to compare the size of the code to the original estimate (if any) when the project was scheduled. If it varies significantly from the estimate, figure out why, so you can learn from your estimation process.

Limit inspection meetings to a maximum of two hours. At the conclusion of the review of each function, decide whether the code should be accepted as is or sent back for rework.

Rework. The author makes all suggested corrections, gets a clean compile (and Lint if used), and sends it back to the moderator.

Follow-up . The moderator checks the reworked code. If the moderator is satisfied, the inspection is formally complete and the code may be tested.

Other points
One hidden benefit of code inspections is their intrinsic advertising value. We talk about software reuse, while all too often failing spectacularly at it. Reuse is certainly tough, requiring a lot of discipline and work. One reason it fails, though, is simply because people aren’t aware of the code. If you don’t know there’s a function on the shelf, ready to rock ‘n’ roll, then there’s no chance you’ll reuse it. The inspection makes more people aware of what code exists.

The literature is full of the pros and cons of inspecting code before you get a clean compile. My feeling is that the compiler is nothing more than a tool, one that very cheaply and quickly picks up the stupid, silly errors we all make. Compile first, and let the tool, rather than expensive people, pick up the simple mistakes.

Along those same lines, I also believe that the only good compile is a clean compile. No error messages. No warning messages. Warnings are deadly when some other programmer, maybe years from now, tries to change a line. When presented with a screenful of warnings, he’ll have no idea if these warnings are normal or a symptom of a problem.

Breaking the cycle
Inspections break the dysfunctional code-compile-debug cycle. We know firmware is hideously complex and awfully prone to failure. It’s crystal clear from data, both quantitative and anecdotal, that code inspections are the cheapest and most effective bug beaters in the known universe. Yet few organizations, especially smaller ones, use them on their firmware.

Inspect all of your code. Make this a habit. Resist the temptation to abandon inspections when the pressure heats up. Being a software professional means we do the right things, all of the time. The alternative is to be a hacker—cranking the code out at will with no formal discipline.

Inspection shouldn’t be limited to code; all specification and design documents benefit from a similar process. esp

Jack G. Ganssle is a lecturer and consultant on embedded development issues. He helps companies with their embedded challenges. Contact him at jack@ganssle.com.

References
For more information, check out the following two books (both very highly recommended). Both are available from the Computer Literacy Book-store and www . amazon.com :

1. Gilb, Tom and Dorothy Graham. Software Inspection. London: TJ Press, 1993.

2. Wheeler, David, Bill Brykczynski and Reginald Meeson. Software Inspection - An Industry Best Practice. Los Alamitos, CA: IEEE Computer Society Press, 1996.

Return to Embedded.com

Send comments to: Webmaster
All material on this site Copyright © 2000
CMP Media Inc. All rights reserved.