Keep It Simple: Towards a Simple Realtime Software Design

Keep It Simple
A perfect design is an enemy of a good design. Often, designers striving fora perfect design may end-up with no design at all due to schedule and costoverruns.  A simple design may not provide best solution to a given problembut it would probably have the best chance of meeting the schedule and costconstraints with acceptable quality. Also, a simple design is easier toimplement, maintain and enhance.
It is better to start with a simple design, making reasonable and rationalcompromises during the design phase. This will save you from making unreasonablecompromises in quality when faced with looming product delivery dates.
Here are some of the "reasonable and rational" designsimplification techniques:
Use lookup tables for complex decision making
Use fixed size arrays whenever possible
Avoid dynamic memory allocation
Reduce the number of tasks in the system
Avoid multi-threaded design
Optimize the design only for the most frequently executed scenarios
Sometimes searching might be more efficient than hashing
Use state machines to simplify design
Use timestamps to avoid running timers
Use object oriented programming
Avoid design hooks for future enhancements
Avoid variable length and bit packed messages
Reduce message handshakes
Simplify the hardware architecture
Prefer general purpose computing platforms over specialized platforms
Do not use proprietary protocols/operating systems
Prefer buy over build for software
Prefer buy over build for hardware
Prefer designs that lead to more reuse
Avoid a heterogeneous hardware and software environment
Consider hardware upgrades to reduce software effort
Minimize configurable system parameters
The "0 or 1 or n" Rule
Many times complex decision making code with cascades of if-then-elsestatements might be replaced with a simple lookup table. A lookup table wouldalso be easier to understand and modify compared to complex and convoluted ifstatements.
Consider the following example, where the system has to support threeterminal types with each terminal type supporting different features. See thefollowing code to see the simplification brought about by lookup tables:
Checking for terminal service types
bool isServiceAllowed(TerminalType type, Service service) { bool allowed = false; if (type == PLATINUM) { if (service == VOICE || service == HIGH_SPEED_DATA || service == FAX) { allowed = true; } } else if (type == GOLD) { if (service == VOICE || service == LOW_SPEED_DATA || service == FAX) { allowed = true; } } else if (type == SILVER) { if (service == LOW_SPEED_VOICE || service == LOW_SPEED_FAX) { allowed = true; } } return allowed; }
Terminal service type check using lookup
bool isServiceAllowed(TerminalType type, Service service) { return (terminalServiceLookup[type][service]); }
top
When managing multiple entities, the simplest possible data-structure isarrays. Consider a case where a certain processor has to support a maximum of1000 user terminals. In this case, we recommend using an array definition for1000 user terminals. This might seem like a waste of memory when the system isactually handling only 100 terminals, but do you really have any use for thatmemory when the system is running at 10% of its rated capacity. A design wherethe system just allocates memory as and when user terminals are added, will befar more complicated to implement. A fixed size array implementation will besimpler to implement and it will probably be more efficient.
top
As far as possible, dynamic memory allocation in Realtime systems should berestricted to message communication related operations. Many times dynamicallocation of objects and structures adds needless complexity to the design.Also with dynamic memory allocation, memory leaks can plague the system for along time. If memory is allocated statically the design remains simple and freeof memory leaks. The designer is freed from thing about when to allocate/freememory and can focus on the core design.
top
Try to reduce the number of tasks in a Realtime system design as it reducesinter-task message interactions. It also eliminates task scheduling delays.Reduced message interaction simplify design as inter-task message handling isreplaced with local function and method invocations. The direct result of thisis reduction in the complexity of the state machines.
top
Scheduling of actions in real-time systems is controlled by state machines.Typically state machines do not call blocking OS primitives while executing astate machine. In such a design, multi-threaded design is of little value.Multi-threaded design complicates the design by introducing synchronizationprimitives to access resources shared among different threads. Multi-threadedprograms are harder to debug than their single-threaded counterparts.
top
Though a programmer has to design a feature for every possible failure andglare scenario, their frequency of occurrence in real-time is low. Thus thedesigners should focus on optimizing the design for the frequent successscenarios. There is no major impact on system performance if the error and glarescenarios are implemented in a simple but non-optimal way.
Consider the call handling inXenon.Here, failure of a digital trunk might be handled by simply searching throughall the call objects for the XEN card and initiating call clears. This couldhave been implemented more efficiently if the digital trunk object maintainedthe list of all of the active calls on that digital trunk. But this failurescenario would not occur frequently. So, the extra complexity of maintaining alist of calls on a per digital trunk basis is not worth the effort.
top
Many times Realtime systems need to maintain hash tables and maps to quicklysearch for an object. Maps and hash tables provide efficient way to dispatchmessages. However when the number of entries in small, a simple search in anarray might be more efficient than incurring the fixed overhead of a hashingalgorithm.
top
Proper use of finite state machines in real-time design avoid unnecessary useof large number of flags and cascaded if statements checking for those flags. Westrongly recommend using state machines as they simplify the code and also makeit easier to understand the feature flow. See the article onhierarchicalstate machines for details.
top
If there is a requirement of keeping track of long time durations, it ispreferable to avoid timer management overheads. This could avoided by taking atime-stamp for the timer-start instant. Then, the time-stamp should besubtracted from the current time to achieve the same effect as timer expiry.This technique is illustrated in the code for handling leaky bucket counters.
Handling Leaky Bucket Counters
bool incrementLeakyCounter() { bool overflow = false; /* This counter needs to be reset if the error counter has not been incremented for a long duration. The main objective is to catch a flurry of errors leading to counter overflow */ if ((currentTime - lastIncrementTime) > LEAK_PERIOD) { counter = 0; } lastIncrementTime = currentTime; counter++; if (counter > OVERFLOW_THRESHOLD) { overflow = true; } return overflow; }
top
Design object oriented as it provides the simplest technique as it mapsdirectly from the problem domain to objects. This simplifies the design cycle asthe design actually mirrors the problem domain. Due to the same reason, objectoriented designs are simpler to understand and maintain  for newdevelopers.
top
It is advisable not to introduce additional complexity in the name of futuredesign hooks. More often than not, these hooks turn out to be a liability thanan asset for the future designers as they might be forced to accept the designbased on these hooks. Thus avoid adding design hooks for the future, as theywill just add additional work that you don't need to do now and will be oflittle help to future designers.
top
Avoid using variable length messages as they add complexity to messageinterface code. Use of variable length fields also forces you to abandon C-structure level definition of messages. This is additional work that you can dowithout.
Same reasoning applies to bit packed messages. Do not define bit packmessages unless mandated by some external protocol that your system supports.Bit packed messages are a constant source of bugs and they lower systemperformance by wasting valuable CPU cycles in message packing and unpacking.
top
When designing message interactions, avoid multiple interactions between twoprocessors. Consider the following example:
XEN card requests a space slot from the CAS card
CAS card allocates the space slot to the XEN and replies with a message.
XEN card now requests the CAS card to route the call
CAS routes the call and replies to the XEN card.
The above interaction could be simplified as:
XEN card requests a space slot and call routing
CAS allocates a space slot and routes the call. CAS replies back to the XEN card.
As you can see above, the four message handshake has been reduced to twomessages only. This reduces one state in the XEN handling of the a call.
top
The first step in keeping the software design is keeping the hardwarearchitecture simple. Software system architects should work closely with thehardware team to simplify the hardware architecture of the product. This mightinvolve:
Reducing the total number of processing nodes in the system.
Simplifying the interconnections between different nodes.
Using off-the-shelf interconnections technologies like Ethernet and TCP/IP.
Using standard PC hardware to implement your system.
Simplifications in hardware directly translate into simplifications insoftware architecture. For example, reducing the number of nodes in the systemwill reduce the number of releases and teams that you need to support. Usingoff-the-shelf interconnection technologies like Ethernet and TCP/IP will opendoors to use of off-the-shelf components like routers and switches.
top
Use a special purpose computing platform only after you have exhausted allpossibilities of using a general purpose platform. For example, if yourapplication requires signal processing capabilities, consider if the performancegoals can be met by a general purpose PC platform without using Digital SignalProcessors (DSPs). General purpose processors might support specializedinstructions that might bring them at par with specialized platforms like DSPs.e.g. the Intel Pentium processors support SSE, SSE2 instructions that can handlereally complex signal processing tasks.
There are several advantages of sticking to general purpose platforms:
Low cost software and hardware development tools.
Its easy to find people with skills in using general purpose platforms.
General purpose platforms have much higher market volume so they are often an order of magnitude cheaper than specialized platforms.
top
Many embedded systems use home grown protocols and operating systems. Thisleads to additional cost to maintain the associated software. Use of standardprotocols and operating systems lowers cost and improves stability of theproduct, as standard products have been subjected to rigorous testing bycountless systems. Proprietary protocols/operating systems often cost a lot moredue to need to train developers.
top
When starting a new software project, consider if you could buy off-the-shelfsoftware modules that can be directly integrated into your system. This savesyou time as you can focus on your core application development, without havingto worry about standard parts like device drivers, external protocols. Whenlooking for external software modules, consider the following factors:
Software you are purchasing is from a reputed company.
Software is supplied along with the source code.
There are no runtime royalties and per product royalties.
The maintenance contract includes clauses on turn around time for critical bugs reported to the company.
Keep in mind that most of the time you can buy software at fraction of thecost of building it in house.
top
Using a off-the-shelf hardware will save you a lot of time and money. Thesoftware team would get the final target hardware on day one of the project.With proprietary hardware, the software team has to invest a lot of developmenteffort in building a test environment on the host machine and finally portingthe software when the hardware becomes available.
top
When developing a hardware and software architecture, prefer designs thatwill reuse already developed software and hardware modules. The futurereusability of the software and modules should also be a factor in choosing newarchitectures. Avoid "lets start with a clean slate" approach to developingsystems. New projects should build over the results of previous project. Thislowers cost by reducing complexity of the system being developed.
top
When developing a new system, use same or similar hardware platform in most of themodules. This will save you the trouble as you do not need to worry aboutbytealignment and ordering and other interoperability issues. Using the same operating system will reducedevelopment effort as you can use common utilities on allmodules in the system. Use of same operating system will also save you money anddevelopment effort as same set of development tools can be used by alldevelopers.
As a corollary to this, consider using the same platform for developersworkstations and the final platform. This will allow developers to use their workstationsfor testing. You would not have to worry about cross development tools and othercosts associated with having a different target platform.
top
Many times you can reduce software complexity by using higher performancehardware platforms. For example:
When faced with a performance crunch, often updating the hardware might be the cheaper option when the costs of developing and maintaining the software optimization are factored in.
Two low performance nodes in the system can be merged into a single high performance node. This will simplify the software design.
It is cheaper to fix the interconnection bottlenecks by updating to the next generation interconnection technology. For example, if a particular link in the system is exceeding the capacity of a 100 Mbps LAN, the cost of using a 1 Gbps LAN will be lower than the cost of optimizing the inter node protocol between the nodes causing the performance bottleneck.
Often, system designers consider software to be "one time" cost and hardwareis considered to be a recurring cost. The designers fail to see that softwaredevelopment and maintenance costs are recurring and will most likely increasewith time. On the other hand, hardware recurring costs will go down with time.
top
Minimize the number of configurable parameters in the system. This willsimplify your system configuration and it will save you coding effort involvedin adding support for change of these parameters.
In our experience most configurable parameters rarely turn out to be useful aftersystem deployment. But these parameters are a big source of system integrationand deployment problems. There are countless horror stories on how wronglyconfigured parameters brought the system down. There are very few cases where aconfigurable parameter saved the day.
Parameters should be made configurable only if they meet any of the followingcriteria:
Site specific tuning is required, i.e. a parameter needs to be assigned different values on different sites.
Operator might need to change the parameter several times during the lifetime of the system and it is not practical to build a software release to make the change.
The product needs to operate in different modes and perform different functions and configurable parameters are required to control this.
We have excluded engineering configurable parameters from the above list. Allparameters that can be finalized during initial field testing should not be madeconfigurable. Making such parameters configurable serves no purpose. Very rarelyyou will encounter a problem that can be fixed just by changing a configurableparameter. Even if you encounter such a problem, the chance that the parameteris actually configurable in your system would be remote.
top
This rule simply states that when designing a system, every feature should bepassed through the following sieves:
Sieve 0: Do we really need this feature?
Sieve 1: OK, we do have to support this feature, but lets support only one instance of this feature.
Sieve n: Looks like multiple simultaneous interactions of this feature need to be supported. Lets design to support "n" instances, even if we need to support just two simultaneous instances.
Try to drop features at Sieve 0. If the features passes through Sieve 0, tryto contain the feature in Sieve 1 and implement only one active instance of thefeature in the system. If the the feature survives Sieve 0 and Sieve 1,implement support for "n" simultaneous instances of the feature in the system.
For example, lets consider the design of a mobile terminal. The marketingfolks request that the mobile terminal should support "broadcast sessions" forvalue added services like TV over a mobile phone. This feature request is firstpassed through Sieve 0 to determine if this is really needed for this terminal.If the features passes through Sieve 0, consider if the mobile terminal shouldonly one broadcast session. If this meets the requirements, implement a systemwith just one broadcast session. This will be a lot simpler than supportingmultiple simultaneously active broadcasts. If you can't get an agreement on onebroadcast, go ahead and implement support for "n" broadcast sessions, eventhough the marketing folks are prepared to testify under oath that they willnever need more than two simultaneous broadcasts.
top
Related Links
Task Design Patterns
Hierarchical State Machines
Object Oriented Design in Embedded Systems
Byte Alignment and Ordering