Computer as a Controller

Introduction

We have built and are now using a real-time platform to test new ideas in control. The benefits of using our own controller rather than a commercial off the shelf (COTS) control module is that it is open. We can try completely new ideas on such an open platform that we develop. With a COTS system, we are limited to what the manufacturer has to offer, or we must remove much of its functionality and build the rest from scratch to our needs. For example, we might purchase a motor position, velocity and torque controller, but to implement our ideas, we may have to use it only in torque control mode.

The computer hardware of the platform we have produced is an industrial PC104 computer from Advantech, of type PCM3350. It is a regular PC but smaller in size, running at 300MHz. Slow in today's desktop standards, but not by much, as far as industrial computers go. On top of the computer are stacked two ISA bus boards, the Kontron ADIO128 AD/DA converter, and MSI-400 quadrature encoder counter. These are directly interfaced to the motors after appropriate power amplification and signal conditioning circuits.

Software running on the computer is mostly written in-house. We are running real-time linux kernel on this system. We built the distribution specifically for this application, from the kernel outwards, using minimal components to keep runtime disk and memory footprint as small as possible. Total size of the operating system including all network connectivity, command line and basic utilities is around 9Mb. It excludes any development software such as compilers. Running off flash memory card, it has no hard disk.

This platform has been extensively tested under the most severe environment; the classroom, for several years with no problems at all.

 

Why Real-Time Linux?

We chose to use real-time linux and an industrial computer in our works for three main reasons.

First, as an experimental platform we need to make changes to the system frequently, which implies it must be flexible, with support of high level languages and services, implying an operating system must be used. We need a computer with more resources this way, which also leads to greater uncertainty in terms of real-time performance  compared to a resource constrained system. However, the flexibility is absolutely necessary. The resources are provided by the PC104 platform and high level services, by the operating system. real-time Linux is suitable for this purpose because it has all of the required services and a common API.

Second, real-time Linux has a good enough timing accuracy for our purpose. We can run a 1kHz control loop with no difficulty.

Third, real-time Linux is open source. This means if there is a problem in the operating system itself, we can examine the source code to fix it, or get help from the open source community.

One drawback is that real-time Linux is no longer free. FSM Labs have closed the source code since version 3.1. However it can still be used as-is. For people starting from scratch, other flavors of real-time linux can be more attractive. Also please keep in mind the new preemptive kernel effort which is trying to reduce the critical sections in the standard 2.6 series Linux kernel and bring it close to a real-time kernel. For coarse grained timing requirements, just this kernel might be enough.

 

What have we done?

The project started as an extracurricular activity in around 2003. A few students started to work with me to develop a computer capable of real-time performance. After reading several books on the subject, reading and testing several open source real-time operating systems, we decided to build a minimal Linux distribution, its kernel patched with real-time Linux. The goal was to make it diskless to reduce size, power consumption and complexity. Therefore a compact flash card was used instead of a hard disk drive (similar methods are now fairly common; see for example, pendrivelinux.com).

After the kernel, we built the directory structure, chose the utility programs and chose the appropriate libraries which they depended on. This is not an easy task as many people upgrading a GNU-Linux system will know; since various utility programs may require different versions of the same library. But they cannot coexist on the same system. Therefore upgrading may create an avalanche effect where upgrading one program might require the administrator to upgrade an old version of a library it depends upon, which then requires the upgrade of other programs in the system which depended on that library. In modern distributions this is automatically checked and taken care of, but in hand-built distributions such as ours, the work required to do a version upgrade may just as well equate complete redesign. Newcomers should take note of this difficulty.

The initial trials were successful when we first booted off the flash disk. However, those machines only worked for three or four days in a row, after which the flash disk burned out. We discovered that the operating system was continuously writing to the disk, which caused the breakdown of the Flash EEPROM cells. (This can now be circumvented by using flash aware file systems and a technique called wear leveling)

As a next step, we designed the system such that it booted off the flash disk, but copied itself into the RAM memory, with the advantages stated above. Since then, the system has worked without any problems, through three years of intense laboratory use for the Embedded Systems Course by uninitiated students. We have never had any operating system related problems using this setup. (A similar method is now directly supported by modern Linux kernels by passing a parameter at boot time)

Almost all the programming work was done by our student Emrah Parlakay.

 

How does it work?

After power on, the operating system boots from compact flash card, into RAM memory, after which flash is unmounted from the system. This allows two things. It is fast since everything is in memory, and second we are immune from unclean disk unmounts such as when power is interrupted or system crashes. One drawback is that there is no non-volatile media but that is simply solved by re-mounting the flash disk to record any data.

At the end of boot, real-time features of the kernel such as scheduler, timer and FIFO buffers, are loaded in the form of modules. Finally we get the normal Linux boot prompt. Now the user can log in (we log in as root without any password in this platform), and start using the system like a normal Linux system with multitasking and complete network support. The hidden extra is, of course, that you now have support to run real-time tasks.

 

How are programs developed and ran?

Programs are developed in a server computer, which is a plain Debian distribution. We write our programs here using the Real-Time Linux API, compile and link them with the necessary real-time Linux libraries, and obtain an object file .o, which is actually a kernel module.

The normal Linux kernel has a monolithic structure meaning that the core of the OS must be a single large executable. A flexibility is provided which allows software modules to be inserted into the kernel using appropriate interfaces placed in at kernel compile time. The general idea is shown in the figure below:

 

Real-time Linux brings in another layer of complexity. It is a dual kernel structure where the real-time kernel sits on top of the computer hardware and controls all of it while scheduling and servicing all real-time programs.  This far it is simple. To manage higher level functions such as file system, network, human interface devices and even a windows interface, the computer still runs the Linux system. However, only as the lowest priority task of real-time Linux, which is actually controlling the whole machine. The two kernels are fused into a single executable. See the figure below:

This is the reason why the user sees a completely normal Linux computer in front of him while he also has, now, real-time services underneath. The non real-time side of the system is called the "user space".

Since the system is not conventional, real-time programs not run in the usual way of simply calling them from the command prompt. To start real-time tasks, the user must compile them as a kernel module, and insert them into the kernel to start them running.

With this setup we can get timing accuracies down to 125us. If we run the same setup on an Intel CPU, we get an order of magnitude better timing (well these are the only computers we have now). So control loops of nearly 100kHz are theoretically possible. This is on a 300MHz processor running a full blown operating system!

 

Quirks

The resulting system is quite stable and easy to use especially after seeing some working examples. However there are some strange sides to writing and running kernel modules especially on a dual kernel system.

Non real-time services

First, real-time kernel does not have high level services such as file system, terminal or network. They are all non-real time services (think of how a keystroke or hard disk access could ever be real-time; there is too much uncertainty). To use such services from a real-time task, we must have a companion task in the non real-time, or "user space" of the system. The two exchange data using a FIFO buffer. We must thus write two separate programs; the real-time side as a kernel module, and the user space side as a normal program; then start them both at the same time.

No awareness of a terminal

Since real-time tasks cannot wait for a non real-time human user, there is no notion of terminal services from a real-time task. They must be accomplished as shown above. You may not pause for a keystroke, or even should not print anything to the screen since it takes an undeterministic amount of time to complete.

No floating point support in the kernel

Since an operating system kernel can be thought of as a huge state machine, the designers have deliberately decided to prevent the use of  floating point arithmetic. However in control applications it is essential to have hardware floating point support. Therefore, provision has been made to enable floating point math using a special command within each task. When the task is killed, we must explicitly remember to turn off this support, or we may cause a crash.

No memory protection

Since our real-time tasks run as kernel modules, memory protection does not apply for them. They are free to access and modify any part of the computer's memory. This is important to keep in mind in a real-time Linux system with non-volatile storage permanently mounted since a runaway program can destroy data or the operating system itself. However, in our case, this is not likely. Any crash may modify RAM contents, but Flash disk is not easily overwritten since it is not mounted. Even if it is, we can simply re-write the original contents and easily start over.

On the other hand, this means we have complete control over the hardware, and writing a device driver becomes much simpler compared with user space.

Linux can be killed

This is an interesting property of the dual kernel structure. We can first start a real-time task by installing its module into the real-time kernel. While it is running, we can shut down the user space linux running on the top. This will not effect our real-time task in any way if it is not using any of the user space services; it will happily continue executing.

This trick is sometimes deliberately used when we need good timing accuracy. By shutting down user side Linux, we are getting rid of its critical sections, causing the improved performance.

No keyboard, no mouse

The system can easily be programmed to start one or more tasks after boot rather than a standard user shell. Since this is an embedded system, a user is not required to start or run it. We only need to power it up. In our platform, we do not have any convenient user interfaces because of this.

 

Photographs

Basic system running off a battery and DC-DC converter off a flash disk. We will need to add input output cards, and perhaps a keyboard or mouse, but only for development!

 

Motor control setup built a long time ago by our PhD candidate Selim Yannier. Real-time control in this system was handled the matchbox size motor controller seen on the right, and computer sent it reference values over the serial port. It booted off a DOS floppy.

CAN Bus application. Computer was used to talk to several CAN Bus motor controllers and send them references.