Please note that as of October 24, 2014, the Nokia Developer Wiki will no longer be accepting user contributions, including new entries, edits and comments, as we begin transitioning to our new home, in the Windows Phone Development Wiki. We plan to move over the majority of the existing entries over the next few weeks. Thanks for all your past and future contributions.

Symbian OS Internals/03. Threads, Processes and Libraries

From Wiki
Jump to: navigation, search
Article Metadata
Created: hamishwillee (17 Jan 2011)
Last edited: hamishwillee (06 May 2013)

by Jane Sales

One of the main causes of the fall of the Roman Empire was that, lacking zero, they had no way to indicate successful termination of their C programs.
Robert Firth

In this chapter, I will look at the entities - that is, threads, processes and libraries - that are concerned with executing code under Symbian OS. I'll begin by examining threads, which are the fundamental unit of execution of Symbian OS.

Because processes and threads touch so many components in Symbian OS, this chapter will refer to many areas that I will explain in more detail in other chapters.


What is a thread?

Under Symbian OS, our definition of a thread is that it is the unit of execution; it is the entity that the kernel schedules, the entity to which the kernel allocates CPU resources. You can think of a thread as a collection of kernel data structures that describe the point a program has reached in its execution. In fact, more accurately, these data structures describe one of the points a program has reached in its execution, because a program can contain more than one thread.

The Symbian OS definition of a process is that it is a collection of threads that share a particular address mapping. In other words, the particular mapping of virtual to physical memory at a particular time depends on the process that is running. Each thread within a process can read and write from any other thread's memory, since they share an address space. But more on this later - for now, let's concentrate on threads.

Nanokernel threads

The nanokernel provides the most basic thread support for Symbian OS in the form of nanokernel threads (which from now on I'll call nanothreads). The nanokernel provides support for the scheduling of nanothreads, and for their synchronization and timing services. The thread services the nanokernel provides are very simple - simpler even than those provided by a typical RTOS. We chose them to be the minimal set needed to support a GSM signaling stack.

Nanothreads only ever run in supervisor mode; they never run in user mode. Because of this, each nanothread needs only a single, supervisor mode stack. The nanokernel keeps the address of the nanothread's stack in the NThread class's iStackBase, and the stack's size in iStackSize. The kernel uses these two variables for initialization and, when an exception occurs, for stack checking to protect against stack overflows.

Each nanothread's member data also contains its last saved stack pointer value, iSavedSP. Whenever the nanothread blocks or is preempted, the scheduler saves the ARM processor's context on the nanothread's stack. Next, the scheduler saves the value of the processor's supervisor-stack-pointer register in iSavedSP. Then the scheduler overwrites the processor's supervisor-stack-pointer register, by loading it from the iSavedSP of the new nanothread. Finally, the scheduler restores the processor's context from the new stack, thus bringing about a thread switch. I will cover scheduling in more detail later.

NThread class

The NThread class is derived from a base class, NThreadBase, which is defined in nk_priv.h. Here is a cut-down version of NThreadBase, showing the main points of interest:

class NThreadBase : public TPriListLink 
enum NThreadState
enum NThreadOperation
TInt Create(SNThreadCreateInfo& anInfo,TBool aInitial);
IMPORT_C void CheckSuspendThenReady();
IMPORT_C void Ready();
void DoCsFunction();
IMPORT_C TBool Suspend(TInt aCount);
IMPORT_C TBool Resume();
IMPORT_C TBool ForceResume();
IMPORT_C void Release(TInt aReturnCode);
IMPORT_C void RequestSignal();
IMPORT_C void SetPriority(TInt aPriority);
void SetEntry(NThreadFunction aFunction);
IMPORT_C void Kill();
void Exit();
void ForceExit();
NFastMutex* iHeldFastMutex;// fast mutex held
NFastMutex* iWaitFastMutex;// fast mutex on which blocked
TAny* iAddressSpace;
TInt iTime; // time remaining
TInt iTimeslice;
NFastSemaphore iRequestSemaphore;
TAny* iWaitObj;// object on which this thread is waiting
TInt iSuspendCount; // how many times we have been suspended
TInt iCsCount; // critical section count
TInt iCsFunction; // what to do on leaving CS:
// +n=suspend n times, 0=nothing, -1=exit
NTimer iTimer;
TInt iReturnValue;
TLinAddr iStackBase;
TInt iStackSize;
const SNThreadHandlers* iHandlers; // + thread event handlers
const SFastExecTable* iFastExecTable;
const SSlowExecEntry* iSlowExecTable; //first entry iEntries[0]
TLinAddr iSavedSP;
TAny* iExtraContext; // coprocessor context
TInt iExtraContextSize; // +ve=dynamically allocated
// 0=none, -ve=statically allocated

You can see that the thread base class itself is derived from TPriListLink - this means that the thread is part of a priority-ordered doubly linked list, which is used in scheduling.

The NThread class itself is CPU-dependent - at the moment three versions of it exist, one for ARM, one for X86 and the other for the emulator. To give you a flavor for the kind of functionality that you find in NThread, here is a cut-down ARM version:

class NThread : public NThreadBase 
TInt Create(SNThreadCreateInfo& aInfo, TBool aInitial);
inline void Stillborn() {}
// Value indicating what event caused thread to
// enter privileged mode.
enum TUserContextType
EContextNone=0, // Thread has no user context
EContextException=1, // HW exception while in user mode
EContextUserInterrupt, // Preempted by interrupt in user mode
EContextUserInterruptDied, // Killed while preempted by interrupt taken in user mode
EContextSvsrInterrupt1, // Preempted by interrupt taken in executive call handler
EContextSvsrInterrupt1Died, // Killed while preempted by interrupt taken in executive call handler
EContextSvsrInterrupt2, // Preempted by interrupt taken in executive call handler
EContextSvsrInterrupt2Died, // Killed while preempted by interrupt taken in executive call handler
EContextWFAR, // Blocked on User::WaitForAnyRequest()
EContextWFARDied, // Killed while blocked on User::WaitForAnyRequest()
EContextExec, // Slow executive call
EContextKernel, // Kernel-side context (for kernel threads)
IMPORT_C static const TArmContextElement* const* UserContextTables();
IMPORT_C TUserContextType UserContextType();
inline TInt SetUserContextType()
{ return iSpare3=UserContextType(); }
inline void ResetUserContextType()
{ if(iSpare3>EContextUndefined) iSpare3=EContextUndefined; }
void GetUserContext(TArmRegSet& aContext, TUint32& aAvailRegistersMask);
void SetUserContext(const TArmRegSet& aContext);
void ModifyUsp(TLinAddr aUsp);
TUint32 Dacr();
void SetDacr(TUint32 aDacr);
TUint32 ModifyDacr(TUint32 aClearMask, TUint32 aSetMask);
void SetCar(TUint32 aDacr);
IMPORT_C TUint32 Car();
IMPORT_C TUint32 ModifyCar(TUint32 aClearMask, TUint32 aSetMask);
#ifdef __CPU_HAS_VFP
void SetFpExc(TUint32 aDacr);
IMPORT_C TUint32 FpExc();
IMPORT_C TUint32 ModifyFpExc(TUint32 aClearMask, TUint32 aSetMask);

Key member data of NThread and NThreadBase

Field Description
iPriority The thread's absolute scheduling priority, between 0 and 63 inclusive.
iNState The state of the thread, that is, ready or blocked. Essentially this determines which queue if any the thread is linked into.
iAttributes Bit mask that determines scheduling policy with respect to the system lock and whether the thread requires an address space switch in general.
iAddressSpace Address space identifier for the thread, used to determine whether the correct address space is already active. The actual value has no significance to the nanokernel, only to the Symbian OS memory model.
iTime Counter used to count timer ticks before thread should yield to next equal priority thread.
iTimeslice Number of low-level timer ticks before thread yields to equal priority threads. If negative, it will not yield unless it blocks.
iRequestSemaphore An NFastSemaphore that we use for general wait for any event. For Symbian OS threads, it serves as the Symbian OS request semaphore.
iSuspendCount Integer, ≤0, which equals minus the number of times a thread has been explicitly suspended.
iCsCount Critical Section Count. Integer, ≥0, which indicates whether the thread may currently be killed or suspended. These actions can only happen if this count is zero, otherwise they are deferred.
iCsFunction Integer that indicates what if any action was deferred due to iCsCount being non-zero. If zero, no action is required; if positive, it equals the number of explicit suspensions that have been deferred; if negative, it indicates that a thread exit has been deferred.
iTimer Nanokernel timer object used to sleep the thread for a specified time and to implement wait-for-event-with-timeout functions.
iExitHandler Pointer to function. If it is not NULL the kernel will call the function when this thread exits, in the context of the exiting thread.
iStateHandler Pointer to function that is called if the thread is suspended, resumed, released from waiting or has its priority changed and the iNState is not a standard nanokernel thread state. Used to implement RTOS emulation layers.
iExceptionHandler Pointer to function called if the thread takes an exception. On ARM, the function is called if a prefetch abort, data abort or undefined instruction trap occurs. The function is always executed in mode_svc1 in the context of the current thread, regardless of the exception type.
iFastExecTable Pointer to table of fast executive calls for this thread.
iSlowExecTable Pointer to table of slow executive calls for this thread - see Section for details.

Nanothread creation

The nanokernel provides the static API below to allow kernel modules outside of the nanokernel to create a nanothread:

NKern::ThreadCreate(NThread* aThread, SNThreadCreateInfo& aInfo)

This function takes an SNThreadCreateInfostructure as a parameter. The caller also passes a pointer to a new NThread object, which it has instantiated beforehand. This must be done because the nanokernel cannot allocate or free memory.


The SNThreadCreateInfo structure looks like this:

struct SNThreadCreateInfo 
NThreadFunction iFunction;
TAny* iStackBase;
TInt iStackSize;
TInt iPriority;
TInt iTimeslice;
TUint8 iAttributes;
const SNThreadHandlers* iHandlers;
const SFastExecTable* iFastExecTable;
const SSlowExecTable* iSlowExecTable;
const TUint32* iParameterBlock;
TInt iParameterBlockSize;
// if 0,iParameterBlock is initial data
// otherwise it points to n bytes of initial data

Key member data of SNThreadCreateInfo

Field Description
iFunction Address of code to execute.
iStackBase Base of stack for new NThread.This must be preallocated by the caller - the nanokernel does not allocate it.
iHandlers Points to the handlers for different situations, namely:
iExitHandler called when a thread terminates execution
iStateHandler called to handle state changes when the iNState is not recognized by the nanokernel (used so OS personality layers can implement new NThread states)
iExceptionHandler called to handle an exception
iTimeoutHandler called when the NThread::iTimertimer expires.


The SNThreadHandlers structure looks like this:

struct SNThreadHandlers
NThreadExitHandler iExitHandler;
NThreadStateHandler iStateHandler;
NThreadExceptionHandler iExceptionHandler;
NThreadTimeoutHandler iTimeoutHandler;

Key member data of SNThreadCreateInfo

Field Description
iParameterBlock Either a pointer to a block of parameters for the thread, or a single 32-bit parameter.
iParameterBlockSize If zero, iParameterBlockis a single 32-bit parameter. If non-zero, it is a pointer.
iFastExecTable Allows personality layer to pass in a pointer to a table of exec calls to be used for this thread.
iSlowExecTable Allows personality layer to pass in a pointer to a table of exec calls to be used for this thread.

Creating a nanothread

Returning to NKern::ThreadCreate(), you can see that the caller passes in the address of the stack that the nanothread will use. This is because the nanokernel cannot allocate memory - so anyone creating a nanothread must first allocate a stack for it.


TInt NThread::Create(SNThreadCreateInfo& aInfo, TBool aInitial)
TInt r=NThreadBase::Create(aInfo,aInitial);
if (r!=KErrNone)
return r;
if (!aInitial)
TUint32* sp=(TUint32*)(iStackBase+iStackSizeaInfo.
TUint32 r6=(TUint32)aInfo.iParameterBlock;
if (aInfo.iParameterBlockSize)
wordmove (sp,aInfo.iParameterBlock,
*--sp=(TUint32)__StartThread; // PC
*--sp=0; // R11
*--sp=0; // R10
*--sp=0; // R9
*--sp=0; // R8
*--sp=0; // R7
*--sp=r6; // R6
*--sp=(TUint32)aInfo.iFunction; // R5
*--sp=(TUint32)this; // R4
*--sp=0x13; // SPSR_SVC
*--sp=0; // R14_USR
*--sp=0; // R13_USR


TInt NThreadBase::Create(SNThreadCreateInfo& aInfo, TBool aInitial)
if (aInfo.iPriority<0 || aInfo.iPriority>63)
return KErrArgument;
new (this) NThreadBase;
iHandlers = aInfo.iHandlers ? aInfo.iHandlers : &NThread_Default_Handlers;
iFastExecTable=aInfo.iFastExecTable ? aInfo.iFastExecTable: &DefaultFastExecTable;
iSlowExecTable=(aInfo.iSlowExecTable ? aInfo.iSlowExecTable: &DefaultSlowExecTable)->iEntries;
// iSpare2 is NThread attributes
if (aInitial)
// now that current thread is defined
return KErrNone;

The static function NKern::ThreadCreate() merely calls NThread::Create() (the first code sample), which then calls NThreadBase::Create() (the second sample). This function sets up various properties of the new thread, including:

  • its stack
  • its priority
  • its timeslice
  • its slow and fast exec tables.

NThreadBase::Create() then puts the new nanothread into the suspended state and returns.

NThread::Create() sets up a stack frame for the new nanothread, first copying over the parameter block (if any) and then creating register values, ready for the thread to pop them. The kernel gives the thread's program counter register (on the stack) the value of _StartThread (see the following code sample). To find out more about where in memory we create the thread's stack, see Chapter 7, Memory Models.


__NAKED__ void __StartThread()
// On entry r4->current thread, r5->entry point, r6->parameter block
asm("mov r0, r6 ");
asm("mov lr, pc ");
asm("movs pc, r5 ");
asm("b " CSM_ZN5NKern4ExitEv);

__StartThread merely assigns some registers, so that iFunction is called with the iParameterBlock function as its argument. If iFunction returns, __StartThreadbranches to Kern::Exit to terminate the thread.

Nanothread lifecycle

Nanothread states

A nanokernel thread can be in one of several states, enumerated by NThreadState and determined by the NThread's iNState member data. I will describe these states below:

Value Description
iNState==EReady Threads in this state are eligible for execution. They are linked into the ready list. The highest priority EReady thread is the one that will actually execute at any given time, unless it is blocked on a fast mutex.
iNState==ESuspended A thread in this state has been explicitly suspended by another thread rather than blocking on a wait object.
iNState==EWaitFastSemaphore A thread in this state is blocked waiting for a fast semaphore to be signaled.
iNState==EWaitDfc The thread is a DFC-handling thread and it is blocked waiting for a DFC to be added to the DFC queue that it is servicing. (For more on DFCs, see Section
iNState==ESleep A thread in this state is blocked waiting for a specific time period to elapse.
iNState==EBlocked The thread is blocked on a wait object implemented in a layer above the nanokernel. This generally means it is blocked on a Symbian OS semaphore or mutex.
iNState==EDead A thread in this state has terminated and will not execute again.
iNState==other If you are writing a personality layer (see Chapter 17, Real Time) then you may choose to allow your nanothreads to have extra states; that is your iNState will be able to take a value other than those above. You must then provide an iStateHandler in your NThread, and then the kernel will call this function if there is a transition in state for this nanothread - if it is resumed, blocked and so on.

Mutual exclusion of nanothreads

Critical sections

A critical section is any sequence of code for which a thread, once it has entered the section, must be allowed to complete it. In other words, the nanokernel must not suspend or kill a thread that is in a critical section.

The nanokernel has sections of code that update kernel global data, and where preemption is enabled. This could lead to problems: imagine that thread MARLOW is in the middle of updating a global data structure, when thread BACON preempts it. Then the global data structure is left in a half-modified state, and if BACON tries to access that data a system crash may well result. We can guard against this scenario by protecting each such global data structure by a mutex, but we need to do more than this.

Imagine this time that, while MARLOW is in the middle of updating the global data structure, BACON preempts MARLOW and suspends or kills it. Let's consider the two situations - suspended or killed - separately.

Assume BACON suspended MARLOW, and that BACON is the only thread that might release MARLOW. MARLOW still holds the mutex protecting the global data structure. So now, if BACON tries to access the same global data structure, a deadlock results, because no other thread will release MARLOW.

Assume BACON killed MARLOW. Now we have a situation similar to our first one - we have left the global data structure in a half-modified state, rendering it unusable by any other thread and having the potential to crash the entire system.

To prevent either of these situations from occurring, we give each thread a critical section count, iCsCount.We increment this every time the thread enters a critical section of kernel code, and decrement it when the thread leaves the critical section.

Then, when thread BACON preempts thread MARLOW, and tries to suspend or kill it, the nanokernel first checks MARLOW's critical section count. If the count is zero, the nanokernel can immediately suspend or kill the thread on behalf of thread BACON.

If MARLOW's critical section count is non-zero, then the nanokernel must delay its actions until MARLOW leaves the critical section. The nanokernel sets a flag in MARLOW's iCsFunction member to indicate

a) That further action is required

b) Whether the thread should suspend or exit.

The nanokernel decrements the thread's iCsCount when the thread leaves the critical section. If the thread's iCsCount becomes zero, then the kernel checks the thread's iCsFunctionto see if further action is required. If iCsFunction is set, then the nanokernel suspends or kills the thread, according to the value placed in iCsFunction. Note that thread BACON, which called Suspend() or Kill(), is not blocked at any stage - it simply carries on executing.

Fast mutexes

But what of the actual mutex used to protect global data structures? Keeping the design goals for the nanokernel as a whole in mind, we derived the following requirements for a fast mutex that would efficiently protect short, critical sections of code:

  1. The mutex must be very fast in the case where there is no contention for the mutex
  2. The mutex must have a low RAM footprint
  3. A thread may not wait on a fast mutex if it already holds a fast mutex (fast mutexes are non-nestable)
  4. A thread may not block or exit while holding a fast mutex.

The nanokernel then ensures that a thread is not suspended or terminated while holding a fast mutex. It does this by treating a nanothread that holds a fast mutex as if it were in a critical section - that is, the nanokernel delays suspending or terminating the nanothread until it releases the fast mutex.

This leaves us with the case in which the thread attempts to exit while holding a fast mutex, for example as a result of taking an exception. In this case, as in the one where a thread attempts to exit while in a critical section, the kernel will fault.

If a nanothread holds a fast mutex, then, on a timeslice, the nanokernel will not schedule another nanothread of the same priority in its place.

This is done to reduce the time spent unnecessarily switching between threads in short critical sections.

How fast mutexes work

Each nanothread has a pointer to the fast mutex currently held by the thread (iHeldFastMutex). We only need one fast-mutex pointer, because, as I said earlier, we chose to design EKA2 with non-nestable fast mutexes. Naturally, the fast mutex pointer is NULL if the nanothread does not hold a fast mutex.

Each nanokernel thread also has a pointer to the fast mutex on which it is currently blocked (iWaitFastMutex). Again, this pointer is NULL if the nanothread is not blocked on a fast mutex.

There are two key elements in the fast mutex class. The first is a pointer to the holding thread (iHoldingThread), which is NULL if the mutex is free. The second is a flag (iWaiting), which indicates either that there was contention for the mutex or that the nanokernel deferred an action (such as suspension, termination or round-robinning) because the mutex was held.

The algorithm for waiting on a fast mutex is:

1.  Lock the kernel
2. IF (iHoldingThread!=NULL)
3. iWaiting = TRUE
4. Current thread->iWaitFastMutex = this
5. Yield to iHoldingThread //return with ints disabled, kernel unlocked
6. Lock the kernel
7. Reenable interrupts
8. Current thread -> iWaitFastMutex = NULL
10. Current thread -> iHeldFastMutex = this
11. iHoldingThread = Current thread
12. Unlock the kernel

[--hamishwillee : Comment from freeman: "I believe in the above code, line 2 should be a while loop like this WHILE( iHoldingThread != NULL) ... Or, if there are more than one thread blocked on this mutex, both will go through line 10 and 12, which is incorrect.""]

If the mutex is free, this simply reduces to two variable assignments, and so is very fast. On single-processor systems (all of them to date!), we further optimize this by disabling interrupts rather than locking the kernel when we check iHoldingThread. It is worth looking carefully at the section of pseudo code between the IF and the ENDIF, lines 2 to 9. You can see that as the kernel blocks the thread, it does not remove it from the ready list. Instead, it performs the Yield to iHoldingThread operation, which immediately switches the context to the thread that holds the mutex. We have to be careful in the case where we are using the moving memory model (see Chapter 7, Memory Models, for more on what this means). The context switch that we have just done does not call the memory model hook provided to allow slow process changes, so the memory model does not get chance to perform any page table manipulations. We cannot allow it to do so, because we want a fast mutex operation, and page table manipulations are usually slow. This means that the kernel doesn't guarantee a user address space to be consistent while the current thread holds a fast mutex. (If we are using the multiple memory model, then all is well, since we can perform the address space change, as in this case it is very fast.)

This scheme also gives us priority inheritance on fast mutexes. This comes about because the blocked thread remains on the ready list, so a reschedule can only be triggered if another thread becomes ready whose priority is at least as great as the highest priority blocked thread (see Section 3.6). So the holding thread can only be scheduled out by a thread whose priority is greater than any thread already on the list - in effect its priority is raised to that of the highest priority blocked thread.

The algorithm for releasing a fast mutex is:

1.  Lock the kernel
2. iHoldingThread = NULL
3. Current thread -> iHeldFastMutex = NULL
4. IF iWaiting
5. iWaiting = FALSE
6. Set TheScheduler.iRescheduleNeededFlag to cause reschedule
7. IF (CurrentThread->iCsFunction && CurrentThread->iCsCount==0)
8. Do critical section exit processing for current thread
11. Unlock the kernel

If iWaiting is NULL, then again this becomes just two variable assignments. And again, on single-processor systems we have optimized this by disabling interrupts rather than locking the kernel while checking iWaiting.

Remember that the nanokernel would have set the iWaiting flag if another nanothread had attempted to acquire the mutex while the first nanothread held it.

The nanokernel would also have set the iWaiting flag if another nanothread had attempted to suspend or kill the first one - in this case it would delay the function to be performed, and keep a record of which function is to be performed later by storing it in iCsFunction.

Finally, the nanokernel would also have set the iWaiting flag if the first nanothread's timeslice had expired, because it will delay the round-robin with other equal priority threads until the fast mutex is released. Have another look at lines 7 and 8 of the pseudo code. They say that the critical section exit processing will be called whenever we are not in a critical section and there is a delayed function to be performed (iCsFunction != NULL). We can reach this point for two reasons. The first is that the thread was executing in a critical section when it was killed or suspended, and now it has exited that critical section. The second is the thread held a fast mutex when it was killed or suspended, and now it has released the fast mutex. The exit processing is the same in both cases.

Nanothread death

You can kill a nanothread by calling NThread::Kill(), but beware - this method is only intended for use by personality layers and the EPOC layer, and should not be called directly on a Symbian OS thread. This is because a Symbian OS thread is an extension of a nanothread, and needs to also perform its own actions on thread death, such as setting the thread exit category and the reason code.

So, we believe that killing threads is not a good thing to do and it is much better to ask them nicely to stop! We have implemented this philosophy in the kernel process's Symbian OS threads. The practical consequence of this is that these threads don't need to enter critical sections - so if you were to kill them at the wrong point, you would have the problems outlined in Section

If the subject thread is in a critical section, then the kernel will not kill it immediately, but will mark it exit pending and kill it once it has exited its critical section.

In either case, the exiting thread first invokes its exit handler, which I'll discuss next.

Exit handler

An exiting thread will invoke its exit handler (iExitHandler) if one exists. The exit handler runs with the kernel unlocked and the exiting thread in a critical section, so it is impossible to suspend the thread while running its exit handler.

The exit handler can return a pointer to a DFC (deferred function call) that will be queued just before the thread actually terminates, that is just before the kernel sets the thread's iNState to EDead. The Symbian OS kernel also uses the DFC to perform final cleanup after the thread has terminated - for example to delete the thread control block.

Threads in the emulator

The Win32 NThreadclass has the following members to add to those of the generic NThreadBase:

Field Description
iWinThread Win32 handle that refers to the host thread that underlies this thread.
iScheduleLock Win32 event object used to block the host thread when not scheduled by the nanokernel. Every nanokernel thread except the current thread will either be suspended in the host OS, in which case iWakeup is EResume or EResumeLocked, otherwise it will be waiting on this event object in its control block.
iDivert A function pointer used to divert a thread from normal return from the scheduler. Diverting threads using a forced change of context in the host OS is unsafe and is only used for forced termination of a thread that is suspended.
iInKernel A counter used to determine if the thread is in kernel mode. A zero value indicates user mode. This state is analogous to being in supervisor mode on ARM - it is used primarily to determine the course of action following an exception.
iWakeup Records the method required to resume the thread when it becomes the current thread.

Thread creation

Instantiating a thread in the emulator is a little more complex than on a target phone, as we must create and initialize a host thread, acquire the resources to control its scheduling, and hand the initial data over to the thread. On the target, we hand the initial data over by directly manipulating the new thread stack before it runs; on the emulator this is best avoided as it involves directly manipulating the context of a live host thread. The nanokernel API allows an arbitrary block of data to be passed to the new thread. Because the new thread does not run synchronously with the create call, but rather when the new thread is next scheduled to run, we must hand over this data before the creating thread returns from NThread::Create().

When creating a new nanokernel thread in the emulator, we create an event object for the reschedule lock, then create a host OS thread to run the NThread::StartThread() method, passing an SCreateThreadparameter block. The parameter block contains a reference to the thread creation information and a fast mutex used to synchronize the data block handover. The handover requires that the new thread runs to copy the data, but it must be stopped again straight afterwards, because when NThread::Create() returns the new thread must be left in a NThread::ESuspended state. Now I'll describe in detail how we achieve this.

We create the new Windows thread, leaving it suspended within Windows. NThread::Create() then marks the new thread as pre-empted (within the nanokernel) so that it will be resumed when the nanokernel schedules it to run. The creating thread then locks the kernel, sets up the handover mutex so that it is held by the new thread, and sets a deferred suspension on the new thread. (The deferred suspension is not immediately effective as the new thread holds a fast mutex.) When the creating thread waits on the same fast mutex, the scheduler runs the new thread from its entry point.

The new thread now initializes. It sets an exception handler for the Windows thread so that the emulator can manage access violations, and then copies the data out of the parameter block. Initialization is then complete so the new thread signals the handover mutex. This action activates the deferred suspension, leaving the new thread suspended as required. Signaling the mutex also wakes up the creating thread, which was blocked on the mutex. The creating thread can now release the mutex again and return, as the new thread is ready to run.

When the new thread is resumed, it invokes the supplied thread entry point function and then exits if that returns.

Thread exit

On target, a nanothread is considered dead when its state is NThread::EDead and after it enters TScheduler::Reschedule for the last time and schedules another thread to run. Because of its state, it will never be scheduled to run again, and its control block can be destroyed.

If the emulator followed this model directly, the Windows thread would be left blocked on its iRescheduleLock. Although the host thread and event object could then be discarded when the thread's control block is destroyed by calling the TerminateThread() function (as in EKA1), Windows does not recommend such use of this function.

Instead, we want the Windows thread to exit cleanly by calling the ExitThread() function. To do this, we ensure that we handle the final thread switch (detected by checking a thread state of EDead) slightly differently. We wake up the next Windows thread to run but do not block the dying thread. Instead, the dying thread releases the thread and event handles before calling ExitThread().

Forced exit - diverting threads

When one thread kills another, the victim has to be diverted from its current activity into the thread exit processing. As we saw before, doing such a diversion by making a forced change of context in Windows can destabilize Windows itself.

The vast majority of situations in which this occurs involve a thread that has voluntarily rescheduled, by calling TScheduler::Reschedule(), rather than being preempted, due to an interrupt. In these situations we can divert safely by making the return path from TScheduler::Reschedule() check to see if a diversion is needed. We use the NThread::iDivert member to record the existence of a diversion on the thread.

This only leaves the rare case where the thread has been preempted, in which case diversion has to involve changing the Windows thread context - there is no other alternative.

This doesn't crash Windows for two reasons:

  1. We don't run the emulator on versions of Windows that suffer like this (such as Windows 98)
  2. We ensure that we don't kill the thread when it is executing code inside a Windows kernel function.

Also we recommend that any threads interacting with Windows should use Emulator::Lock() and Unlock() to prevent preemption whilst inside Windows.

Symbian OS threads

The Symbian OS kernel builds on nanothreads to provide support for user-mode threads that are used by standard Symbian OS user applications. We represent these threads using the DThread class.

Earlier in this chapter, I said that nanothreads always execute in supervisor mode and have a single supervisor-mode stack. Each Symbian OS thread object contains a nanothread object; this nanothread is what the kernel actually schedules. The nanokernel cannot allocate memory, so when it is creating a Symbian OS thread, the Symbian OS kernel must allocate the memory for the nanothread's supervisor stack before it makes the ThreadCreate() call to the nanokernel.

Each user-mode Symbian OS thread naturally has a user-mode stack, so, to spell it out, user-mode Symbian OS threads each have two stacks (Figure 3.1). The supervisor-mode stack associated with the nanothread is used by kernel-side code run in the thread's context - that is, the system calls made by the thread.

The use of per-thread supervisor stacks is a fundamental difference from EKA1, in which each thread had only one stack. By adding the supervisor stack to each thread, we allow kernel-side code to be preempted and thus achieve low thread latencies. You can read more about this in Section

EKA2 also provides each Symbian OS thread with:

  • Access to a set of executive functions to enable user-mode threads to gain access to kernel functionality
  • An exception handler, which enables the kernel to handle exceptions in user-mode threads (typically caused by programming errors)
  • An exit handler, which allows resources to be freed and other threads to be notified when a thread terminates.
Figure 3.1 Stacks for a user-mode Symbian OS thread

Symbian OS thread creation

Creating a thread: 1

User-mode code creates a Symbian OS thread by calling RThread::Create(). By default, this method creates a thread belonging to the current process, and returns an open handle to the newly created thread back to the caller.

The RThread::Create() method goes via an exec call (see Chapter 5, Kernel Services) to the kernel, where it ends up inside the method DThread::Create(SThreadCreateInfo&aInfo).

SThreadCreateInfo The basic SThreadCreateInfostructure looks like this:

struct SThreadCreateInfo
TAny* iHandle;
TInt iType;
TThreadFunction iFunction;
TAny* iPtr;
TAny* iSupervisorStack;
TInt iSupervisorStackSize;
TAny* iUserStack;
TInt iUserStackSize;
TInt iInitialThreadPriority;
TPtrC iName;
TInt iTotalSize; // Size including any extras

Key member data of SThreadCreateInfo

Field Description
iHandle Handle on thread, returned to caller.
iType Type of thread (EThreadInitial, EThreadSupervisor, EThread-MinimalSupervisoror EThreadUser). Indicates whether thread can execute in user mode and whether creation should be fast or normal.
iFunction Where to begin execution of this thread.
iPtr The pointer passed as an argument to iFunction.
iSupervisorStack Pointer to supervisor stack. If 0, a stack will be created for it.
iSupervisorStackSize Size of supervisor stack. Zero means to use the default value of 4 KB.
iUserStack Pointer to user stack. This is returned to caller, not passed in.
iUserStackSize Size of user stack. Zero means to use the default value of 8 KB.
iInitialThreadPriority Initial priority for this thread.
iName Name of the thread. If this is non-zero then the thread is a global object.
iTotalSize Size including any extras. This must be a multiple of 8 bytes.

SStdEpocThreadCreateInfo The basic SThreadCreateInfostructure is then derived by SStdEpocThreadCreateInfo to provide three more fields, like so:

struct SStdEpocThreadCreateInfo : public SThreadCreateInfo
RAllocator* iAllocator;
TInt iHeapInitialSize;
TInt iHeapMaxSize;
TInt iPadding; // Make size a multiple of 8 bytes

This structure adds a pointer to the RAM allocator and an initial and a maximum size for the heap. It forms the control block that is pushed onto a thread's stack before that thread starts. The extra fields are used by the standard entry point function to set up the thread's heap - the kernel does not use them itself.

We chose to derive from SThreadCreateInfo so that we could support other kinds of threads in personality layers. Those new types of thread would probably need to pass different parameters to the thread's entry point instead. The authors of the personality layer can do this easily, by deriving a new class from SThreadCreateInfo.

Creating a thread: 2

The DThread constructor sets the state of the new thread to ECreated. The thread stays in this state throughout the construction process, then, once the thread is completely constructed, the kernel changes its status to EReady. This ensures that there are no anomalies arising from the death of a partly created thread.

Then the DThread::Create() method creates a user stack for the thread. To find out more about where in memory we create the thread's stack, see Chapter 7, Memory Models.

Next DThread::Create() calls DThread::DoCreate(). This method continues the work of setting up the thread, calling NKern::ThreadCreate() to create the nanokernel portion of the Symbian OS thread.

On return from DoCreate(), DThread::Create() adds the thread to the object container for threads (assuming it is neither the initial thread, nor a minimal supervisor thread). For more on object containers, see Chapter 5, Kernel Services.

When the Symbian OS thread first starts to run, it executes __StartThread. This calls DThread::EpocThreadFunction(), which checks to see if the thread is a kernel or user thread.

If it is a kernel thread, then DThread::EpocThreadFunction() calls the thread function - this is the function that the caller specified in Kern::ThreadCreate().

If it is a user thread, then the thread creation information is copied to the user stack. Then the CPU is switched to user mode and the process's entry point is called, with a parameter that is either KModuleEntryReasonProcessInit for the first thread in the process, or KModuleEntryReasonThreadInit for subsequent threads.

The process's entry point is __E32Startup in e32\euser\epoc\arm\uc_exe.cia. This code compiles into eexe.lib - Symbian's equivalent of crt0.obj on Windows.

The process entry point then calls RunThread(), which calls UserHeap::SetupThreadHeap(). This is the function that will create the thread's heap if required.

Then, if this is the first thread in the process, the constructors for static data are called before calling E32Main(). Otherwise, the thread function is called straight away.

Over-riding the Symbian OS allocators

You may have noticed that EKA2 does not create the thread's heap. Instead, threads create their own heaps (or share an existing heap) when they first run. This makes it possible for the process itself to hook in and over-ride the normal heap creation function. In this way, you can choose to make a particular process use a memory allocator other than the standard one that Symbian provides in EUSER. You can also insert additional heap tracking or diagnostic functions. However, you should note that the function UserHeap::SetupThreadHeap() must be in a static library if you want to automatically over-ride heap creation in a process; the code for the RAllocator-derived class can be in a DLL. You do need to be careful when over-riding User-Heap::SetupThreadHeap(), because it is called before static data is initialized.

The DThread class

As we've already seen, the DThread class represents a Symbian OS thread. Now let's find out a little more about it. DThread derives from DObject, which makes it a dynamically allocated reference counted object inside the kernel (see Chapter 5, Kernel Services, for more on this). The DThread has an embedded nanothread (iNThread), which enables the nanokernel to schedule it.

We then derive the DThread class further to give a concrete CPU/MMU specific class - on ARM CPUs this is called DArmPlatThread. DArmPlatThread contains some CPU specifics but in general, it does not add much functionality to DThread.

Here is a cut-down version of the DThread class to give you a flavour for the kind of thing it includes:

class DThread : public DObject
enum {EDefaultUserTimeSliceMs = 20};
enum TThreadState
enum TOperation
void Destruct();
TInt Create(SThreadCreateInfo& aInfo);
TInt SetPriority(TThreadPriority aPriority);
IMPORT_C void RequestComplete(TRequestStatus*& aStatus,
TInt aReason);
IMPORT_C TInt DesRead(const TAny* aPtr, TUint8* aDes, TInt aMax, TInt aOffset, TInt aMode);
IMPORT_C TInt DesWrite(const TAny* aPtr, const TUint8* aDes,TInt aLength, TInt aOffset, TInt aMode,
DThread* aOriginatingThread);
// not memory model dependent
TInt DoCreate(SThreadCreateInfo& aInfo);
IMPORT_C void SetThreadPriority(TInt aThreadPriority);
void SetDefaultPriority(TInt aDefaultPriority);
void AbortTimer(TBool aAbortAbsolute);
void Suspend(TInt aCount);
void Resume();
void ForceResume();
void Exit();
void Die(TExitType aType, TInt aReason,
const TDesC& aCategory);
TInt Logon(TRequestStatus* aStatus, TBool aRendezvous);
void Rendezvous(TInt aReason);
// memory model dependent
TInt AllocateSupervisorStack();
void FreeSupervisorStack();
void FreeUserStack();
TInt AllocateUserStack(TInt aSize);
TInt RawRead(const TAny* aSrc, TAny* aDest, TInt aLength,TInt aFlags);
TInt RawWrite(const TAny* aDest, const TAny* aSrc,TInt aLength, TInt aFlags, DThread* aOriginatingThread);
DChunk* OpenSharedChunk(const TAny* aAddress, TBool aWrite,TInt& aOffset)
static void DefaultUnknownStateHandler(DThread* aThread,TInt& aOperation, TInt aParameter);
static void EpocThreadFunction(TAny* aPtr);
static TDfc* EpocThreadExitHandler(NThread* aThread);
static void EpocThreadTimeoutHandler(NThread* aThread,TInt aOp);
TUint32 iIpcCount;
TLinAddr iUserStackRunAddress;
TInt iUserStackSize;
TUint32 iFlags;
DProcess* iOwningProcess;
SDblQueLink iProcessLink;
TInt iThreadPriority;
DObjectIx* iHandles;
TUint iId;
RAllocator* iAllocator;
RAllocator* iCreatedAllocator;
TTrap* iFrame;
TTrapHandler* iTrapHandler;
RArray<STls> iTls;
CActiveScheduler* iScheduler;
TExceptionHandler iExceptionHandler;
TUint iExceptionMask;
TExcTrap* iExcTrap;
TInt iDebugMask;
TThreadMessage iKernMsg;
DObject* iTempObj;
DObject* iExtTempObj;
TAny* iTempAlloc;
SDblQue iOwnedLogons;
SDblQue iTargetLogons;
RMessageK iSyncMsg;
TDfc iKillDfc;
SDblQue iClosingLibs;
TPriListLink iWaitLink;
TInt iDefaultPriority; // default scheduling priority
TAny* iWaitObj; // object on which this thread is waiting
// handler for extra thread states - used by RTOS
// personality layers
TUnknownStateHandler iUnknownStateHandler;
// pointer to extra data used by RTOS personality layers
TAny* iExtras;
TAny* iSupervisorStack;// thread’s supervisor mode stack
TInt iSupervisorStackSize;
TUint8 iSupervisorStackAllocated;
TUint8 iThreadType;
TUint8 iExitType;
TUint8 iPad1;
TInt iExitReason;
TBufC<KMaxExitCategoryName> iExitCategory;
// list of held mutexes, used only for acquisition
// order checking
SDblQue iMutexList;
// things to clean up when we die
TPriList<TThreadCleanup,KNumPriorities> iCleanupQ;
TTimer iTimer;
NThread iNThread;

Key member data of DThread

Field Description
iFlags Thread flags (system critical, last chance, process permanent, original).
iExitType Top level thread exit reason (kill, terminate or panic) or EExitPending if thread still running.
iExitReason Exit code (return value from thread main function or reason supplied to kill, terminate or panic call).
iExitCategory String providing additional exit information in case of panic.
iThreadType Type of thread: EThreadInitial, EThreadSupervisor, EThreadMinimalSupervisoror, or EThreadUser.
iOwningProcess Pointer to DProcessobject that represents the process to which this thread belongs.
iThreadPriority Priority of this thread in either absolute or process-relative form. Values between 0 and 63 represent absolute priorities; values between -8 and-2 represent priorities relative to that of the owning process.
iHandles Pointer to array (DObjectIx) of thread-local handles for this thread.
iId Symbian OS thread ID of this thread (unsigned 32-bit integer).
iAllocator Pointer to the user-side heap being used by this thread. This is stored by the kernel on behalf of the user code and is not used by the kernel itself.
iExceptionHandler Pointer to this thread's user-side exception handler. This will be invoked in user mode in the context of this thread if a handled exception occurs.
iExceptionMask Bit mask indicating which types of exception the user-side exception handler handles. These are defined by the enum TExcType.
iExcTrap Pointer to the thread's exception trap handler. The handler will be invoked if an exception occurs in the context of this thread. The trap part of the handler provides the ability to return to the beginning of a protected code section with an error code. NULL if no such handler is currently active.
iDebugMask Per-thread debug mask. This is ANDed with the global debug mask to generate the debug mask used for all kernel tracing in the context of this thread.
iKernMsg Kernel-side message sent synchronously to kernel-side threads. Used for communication with device driver threads.
iOwnedLogons Doubly linked list of thread logons owned by this thread (that is, this thread is the one which requests notification of another thread terminating).
iTargetLogons Doubly linked list of thread logons whose target is this thread (that is, another thread requests notification if this thread terminates).
iSyncMsg Symbian OS IPCmessage (RMessageK) reserved for sending synchronous IPC messages.
iKillDfc The Symbian OS exit handler returns a pointer to this DFC when the thread terminates. The nanokernel queues the DFC immediately before terminating the thread. The DFC runs in the context of the supervisor thread and is used to clean up any resources used by the thread, including the DThread object itself.
iClosingLibs This is a doubly linked list holding a list of DLibrary objects which require user-side destructors to be called in the context of this thread.
iMState This is actually the iSpare1field of iWaitLink. It indicates the state of this thread with respect to Symbian OS wait objects.
iWaitLink This is a priority queue link field used to link the thread on to the wait queue of a Symbian OS wait object. Note that the priority of this link should always be equal to the thread's nanokernel priority.
iWaitObj This is a pointer to the Symbian OS wait object to which this thread is currently attached, NULL if the thread is not attached to any wait object.
iSupervisorStack Base address of thread's supervisor mode stack.
iCleanupQ Priority-ordered list (64 priorities, same structure as scheduler) used to hold callbacks to be invoked if the thread terminates. Also used to implement priority inheritance - adding a high-priority cleanup queue entry will raise a thread's priority. Thread scheduling priority is calculated as the maximum of iDefaultPriority and the highest priority of any entry in the cleanup queue.
iNThread The nanokernel thread control block corresponding to this thread.

Types of Symbian OS thread

There are four types of Symbian OS thread, which are determined by the iThreadType field in DThread. This takes on one of the values in the enumeration TThreadType.

Value Description
iType==EThreadInitial There is only ever one initial thread in the system, and this is the first thread to run on a device at boot time. This thread's execution begins at the reset vector and eventually becomes the null thread, which is the thread with the lowest possible priority in the system. For more on Symbian OS bootup, see Chapter 16, Boot Processes.
iType==EThreadSupervisor Supervisor threads run only in supervisor mode, never in user mode. The memory model allocates a supervisor stack for these threads; you may vary its size by passing in a parameter during thread creation. Usually it is best to pass in a value of 0, which tells the kernel to use the default size of 4 KB (one page). Supervisor threads have time slicing enabled by default, with a timeslice of 20 ms.
iType==EThreadMinimalSupervisor These threads are intended for use by RTOS personality layers and are similar to supervisor threads. The key requirement for an RTOS threads is a fast creation time, so the kernel does not give these threads.

The memory model can allocate the supervisor stack just as it does for supervisor threads, but if you wish, you can preallocate an area of memory and pass a pointer to it when you create the thread. Finally, RTOS threads generally don't need time slicing, so we disable it by default.

iType==EThreadUser These are the threads used to run standard user applications. They run in user mode for most of the time, although they do run in supervisor mode during executive calls.

As we have seen, these threads have two stacks, a user-mode one and a supervisor-mode one; the memory model allocates both of these dynamically. The memory model allocates the supervisor stack in the same way as for supervisor threads, with a default stack size of 4 KB. It allocates the user stack during thread create time; it creates the user stack in the address space of the process to which the thread belongs. (I will discuss this in more detail in Chapter 7, Memory Models.) The creator of a user thread can specify the size of the user stack at creation time. User threads have time slicing enabled by default, with a timeslice of 20 ms.

Identifying Symbian OS and personality layer threads

The easiest way to determine if a given NThread is a Symbian OS thread or belongs to an executing personality layer is to examine the nanothread's handlers:

// additional thread event handlers for this NThread
SNThreadHandlers* NThreadBase::iHandlers()

The function I use looks like this:

SNThreadHandlers *gEpocThreadHandlers;
// Grab the thread handlers from a thread that we
// know is a Symbian OS thread.
// The extension and DLL loader is guaranteed to be
// a Symbian OS thread, so we can use an extension
// entry point to capture its thread handlers.
// Extension entry point
. . .
// Get Symbian OS thread from a NThread
// if there isn’t one, it’s a personality
// layer thread, return NULL.
DThread *GetEpocThread(NThread *aNThread)
if (aNThread->iHandlers != gEpocThreadHandlers)
return NULL; // personality layer thread
DThread* pT = _LOFF(aNThread, DThread, iNThread);
return pT;

The method I use is quite simple. First, in a global variable I save a pointer to the handlers of the extension and DLL loader (a thread that is known to be a Symbian OS thread). Then, later, I compare my thread's nanothread's iHandlers pointer, and if it is the same as the global variable, then my thread is a Symbian OS thread too.

Symbian OS thread lifecycle

Symbian OS thread states

Each Symbian OS thread has a state, known as the M-state. (This is in addition to the N-state of each Symbian OS thread's embedded nanothread.) When the Symbian OS thread waits on or is released from a Symbian OS semaphore or mutex, its M-state changes.

The M-state may take on any of the values in the enumeration TThreadState - a complete list follows. An RTOS may add more M-states; I'll discuss this further in the next section,

Field Description
iMState==ECreated This is the initial state of all Symbian OS threads. It is a transient state - the kernel puts the thread into this state when it creates the DThread control block and keeps the thread in this state until the kernel is ready to resume it, at which point it changes the state to EReady.
iMState==EDead This is the final state of all Symbian OS threads. A thread enters this state when it reaches the end of its exit handler, just before the nanokernel terminates it.
iMState==EReady A thread is in this state when it is not waiting on, or attached to, any Symbian OS kernel wait object (semaphore or mutex). This state does not necessarily imply that the thread is actually ready to run - this is indicated by the N-state. For example, a thread that is explicitly suspended or waiting on a nanokernel wait object (generally a fast semaphore) has iMState==EReady, if it is not attached to any Symbian OS wait object.
iMState==EWaitSemaphore This state indicates that the thread is currently blocked waiting for a Symbian OS semaphore and is enqueued on the semaphore's wait queue. The thread's iWaitObj field points to the semaphore.
iMState==EWaitSemaphoreSuspended This state indicates that another thread has explicitly suspended this thread after it blocked on a Symbian OS semaphore and was enqueuedon the semaphore's suspended queue. The thread's iWaitObj field points to the semaphore.
iMState==EWaitMutex This state indicates that the thread is currently blocked waiting for a Symbian OS mutex and is enqueued on the mutex wait queue. The thread's iWaitObj field points to the mutex.
iMState==EWaitMutexSuspended This state indicates that another thread has explicitly suspended this thread after it blocked on a Symbian OS mutex and was enqueued on the mutex suspended queue. The thread's iWaitObj field points to the mutex.
iMState==EHoldMutexPending This state indicates that the thread has been woken up from the EWaitMutex state but has not yet claimed the mutex. The thread is enqueued on the mutex pending queue and the thread's iWaitObj field points to the mutex.
iMState==EWaitCondVar This state indicates that the thread is waiting on a condition variable. The thread is enqueued on the condition variable's wait queue, iWaitQ, and its iWaitObj field points to the condition variable.
iMState==EWaitCondVarSuspended This state indicates that the thread has been suspended while waiting on a condition variable. The thread is removed from the condition variable's iWaitQ, and enqueued on iSuspendQ. The thread's iWaitObj field points to the condition variable.

M-state changes

A thread's M-state can change because of any of the following operations:

  1. The thread blocks on a wait object
  2. The thread is released from a wait object
  3. The thread is suspended
  4. The thread is resumed
  5. The thread's priority is changed. This can cause a transition from EWaitMutex to EHoldMutexPending if the mutex is free and the thread's priority is increased
  6. The thread is killed.

Multiple state changes can occur in this case as the thread proceeds through the exit handler. The first state change will occur as a result of a ReleaseWait() call at the beginning of the exit handler. This call cancels the thread's wait on any Symbian OS wait object and detaches the thread from the wait object, that is it removes it from any queues related to the wait object. The final state change will be to the EDeadstate at the end of the exit handler.

The first five of these operations are protected by the system lock mutex. In the case of thread exit, the initial call to make the thread exit is protected by the system lock, as is the ReleaseWait() call, but the exit handler runs without the system lock for some of the time.

Other M-states

RTOS personality layers can add new M-states to indicate that threads are waiting on non-Symbian OS wait objects. To make this easier to implement, each Symbian OS thread has an unknown state handler, iUnknownStateHandler. Let's see how it works.

Assume that a thread is in an M-state unknown to the kernel (that is, not one of those I have discussed above). Then the kernel will call the unknown state handler after the kernel suspends, resumes, kills or changes the priority of that thread. The unknown state handler can then adjust the RTOS wait object's queues and transition the M-state if necessary.

The unknown state handler is not involved in all state transitions: the RTOS personality layer code will block and release threads from an RTOS wait object directly.

It is worth noting that the Symbian OS thread unknown state handler, iUnknownStateHandler, is completely different to the nanokernel thread unknown state handler, iNThread->iHandlers->iStateHandler. Which you use depends on exactly how you implement your RTOS personality layer. Clearly there are two choices:

  1. Over the nanokernel: This is the usual method, described in Chapter 17, Real Time. Personality layer threads are bare NThreadsand you use NThread::iHandlers->iStateHandler to add extra wait states for new wait objects.
  2. Over the Symbian OS kernel: In this case, personality layer threads are Symbian OS kernel threads, and DThread::iUnknownStateHandler would be used to add extra wait states for new wait objects. The advantage of using this method is that you can make use of Symbian OS kernel services in your personality layer threads (for example semaphores, mutexes and memory allocation). The disadvantage is that Symbian OS threads use a lot more memory - over 800 bytes per DThread, plus 4 KB of stack.

Cleanup queues

Each Symbian OS thread has a cleanup queue, iCleanup. This queue holds thread cleanup items, which are TThreadCleanup-derived objects.


class TThreadCleanup : public TPriListLink
IMPORT_C TThreadCleanup();
void ChangePriority(TInt aNewPriority);
IMPORT_C void Remove();
virtual void Cleanup()=0;
DThread* iThread;
friend class Monitor;

The cleanup queue is a priority queue, with each of the cleanup items on it having a priority between 0 and 63. Each cleanup item also has a callback function. When the Symbian OS thread terminates, its thread exit handler calls the callback functions in descending order of priority - it always holds the system lock while it does this. It is, however, permissible for a cleanup item's callback to release the system lock, for example to delete the cleanup item itself or to perform some other long-running operation. (The kernel also locks the system when it modifies the cleanup queue.) The kernel sets the priority of cleanup items that are used purely to provide notification of thread exit to zero. Currently the Symbian OS kernel uses thread cleanup items for two purposes:

1. It associates a TThreadMutexCleanup object with every Symbian OS mutex. The kernel adds this item to the cleanup queue of the mutex-holding thread so that it can release the mutex if the thread terminates

2. The kernel uses cleanup items with non-zero priority to implement priority inheritance for Symbian OS mutexes; it does this by adjusting the priority of the TThreadMutexCleanup item.

Priority inheritence

How does this priority inheritence work? We define the priority of a Symbian OS thread to be the maximum of its own priority, and that of any cleanup item on its queue. Now suppose that a low-priority thread owns a mutex when a higher-priority thread comes along and blocks on that mutex. The kernel will then adjust the priority of the cleanup item associated with the mutex to be equal to the priority of the high-priority thread. Thus the low-priority thread's priority, which is the maximum of its own priority and any cleanup item it owns, is also boosted, since it owns the cleanup item associated with the mutex. If there are no higher-priority ready threads in the system, it will continue to run until it releases the mutex, at which point its priority drops to its normal level.

Notification of device driver client termination

There is a third use for thread cleanup items: device drivers can use them to get notification of their clients terminating unexpectedly. For example, the local media sub-system uses this mechanism to ensure that the resources held by the driver on behalf of the client thread are cleaned up if the client thread terminates. It has to use this method, since multiple threads in the same process may use a single logical channel to communicate with the driver, meaning that the channel Close() method will not necessarily be called when a client thread terminates.

Thread logon and rendezvous

Thread logon is a mechanism that allows a thread to request notification that another thread has reached a given point in its execution or has terminated. The first form of notification is known as a rendezvous.

Each thread has two doubly linked lists: one of target logons (iTargetLogons - representing threads which are requesting notification of a rendezvous with or the termination of this thread) and one of owned logons (iOwnedLogons - representing requests by this thread to be notified of a rendezvous with other threads or when other threads terminate). Rendezvous requests are stored at the head of iOwnedLogons and logon (termination) requests are stored at the tail of this list.

The kernel handles process rendezvous and logon using an identical mechanism.

The TLogon::LogonLock fast mutex protects all these lists (over all the threads and processes in the system). When a rendezvous is signaled by a thread, the target logon list is iterated from the head and all rendezvous requests completed. When a thread terminates, the thread exit handler completes all the logons on its target logon list (both rendezvous and termination) with the exit reason and discards any remaining logons on its owned logon list.

We use the iExiting field to prevent a new logon from being added to a thread after it has completed all its target logons, which would result in the new logon never being completed. The iExiting flag is set to ETrue at the beginning of the thread exit handler. Any attempt to add a new logon to (or rendezvous with) a thread whose iExiting flag is set will fail and will complete immediately with KErrDied.

Since thread logons need notification of thread exit just as cleanup items do, why do we need a separate mechanism for them rather than just adding them to the thread cleanup list? This is because the kernel protects thread cleanup queues with the system lock held and so they must be fast, with a bounded execution time. But, of course a thread may have any number of logons outstanding, and canceling a logon requires the kernel to walk the list of outstanding logons. This means that the time for which the system lock could be held is unbounded.

You might be wondering why this is so. After all, the kernel releases the system lock after it processes each cleanup item. However, the situations are subtly different. The kernel processes cleanup items when a thread exits, walking the queue and processing each cleanup item. However, rather than searching for and removing a particular item, it simply wants to remove all items, so it can use an algorithm like this one:

// remove the first item (if any) from the list
// and return a pointer to it.
// p is null when there are no more items left
p = GetFirstItem();
if (p)

But when attempting to cancel a logon, the kernel needs to search the list for a particular item and remove just that one item. If it releases the lock at any point during this scan it can lose its place - the last item it checked may be removed from the list and placed on another list, or even destroyed, before this portion of code runs again. So the kernel must hold the lock throughout the scan, and this means that it cannot use the system lock.

We also considered the fact that processes don't have cleanup queues. Once we had taken this into account, it made sense to create an entirely new mechanism for thread logons, knowing we could re-use it for process logons. So, we use a separate queue for logons, and we protect this queue with a different mutex.

Symbian OS thread-synchronization objects

The Symbian OS kernel provides support for more complex thread-synchronization objects than the nanokernel does. These objects are Symbian OS semaphores and mutexes.

Symbian OS semaphores are standard counting semaphores which support multiple waiting threads (unlike nanokernel fast semaphores) and which release waiting threads in priority order.

Symbian OS mutexes are fully nestable - a thread can hold several mutexes at once and can also hold the same mutex several times. They support priority inheritance - the holding thread inherits the priority of the highest-priority waiting thread if that is higher than the holding thread's usual priority. The kernel and memory model use Symbian OS mutexes extensively to protect long-running critical code sections.

Semaphores - DSemaphore

Symbian OS semaphores are standard counting semaphores. The semaphore maintains a count: if the count is positive or zero, no threads are waiting; if it is negative, the count is equal to minus the number of waiting threads.

There are two basic operations on semaphores:

  • WAIT. This decrements the count atomically. If the count remains nonnegative the calling thread continues to run; if the count becomes negative the calling thread is blocked
  • SIGNAL. This increments the count atomically. If the count was originally negative, the kernel releases the next waiting thread.

The kernel protects Symbian OS semaphore operations by the system lock.

It is important to note that DSemaphore operations rely on fields that are present in DThread but not in NThread. This means that only Symbian OS threads may invoke Symbian OS semaphore operations - it is not permitted for an IDFC or a non-Symbian OS thread to signal a Symbian OS semaphore.

We use the DSemaphore class to represent a Symbian OS semaphore. This class is derived from DObject, which makes it a dynamically allocated reference counted object. DSemaphore is the kernel object referred to by a user-side RSemaphore handle:

class DSemaphore : public DObject
TInt Create(DObject* aOwner, const TDesC* aName,
TInt aInitialCount, TBool aVisible=ETrue);
void WaitCancel(DThread* aThread);
void WaitCancelSuspended(DThread* aThread);
void SuspendWaitingThread(DThread* aThread);
void ResumeWaitingThread(DThread* aThread);
void ChangeWaitingThreadPriority(DThread* aThread, TInt aNewPriority);
TInt Wait(TInt aNTicks);
void Signal();
void SignalN(TInt aCount);
void Reset();
TInt iCount;
TUint8 iResetting;
TUint8 iPad1;
TUint8 iPad2;
TUint8 iPad3;
SDblQue iSuspendedQ;
TThreadWaitList iWaitQ;
friend class Monitor;

Key member data of DSemaphore

Field Description
iCount The semaphore count.
iResetting A flag set while the semaphore is being reset; this occurs just prior to the semaphore being deleted and involves releasing any waiting or suspended threads. The flag is used to prevent any more threads from waiting on the semaphore.
iSuspendedQ This is a doubly linked list of threads which are both waiting on the semaphore and explicitly suspended. These threads will have iWaitObj pointing to this semaphore and will have M-state EWaitSemaphore-Suspended.
iWaitQ A list, in decreasing priority order, of threads that are waiting on the semaphore and not explicitly suspended. Note that this is a difference from EKA1 - under the old version of Symbian OS, the kernel released threads in the order that they had waited on the semaphore. Threads in this list will have their iWaitObj pointing to this semaphore and their M-state will be EWaitSemaphore.

Note that the iWaitObj field under discussion here is the DThread member, not the NThread member of the same name. Threads that are explicitly suspended as well as waiting on a semaphore are not kept on the semaphore wait queue; instead, the kernel keeps them on a separate suspended queue, iSuspendedQ, which is just a standard doubly linked list. We do not regard such threads as waiting for the semaphore - if the semaphore is signaled, they will not acquire it and the semaphore count will simply increase and may become positive. To acquire the semaphore, they must be explicitly released, at which point the kernel removes them from iSuspendedQ and adds them to iWaitQ.

Mutexes - DMutex

Symbian OS mutexes provide mutual exclusion between threads, but without the restrictions imposed by the nanokernel mutex. So,

  • It is possible to wait on a Symbian OS mutex multiple times, provided it is signaled the same number of times
  • It is possible to hold several Symbian OS mutexes simultaneously, although care is required to avoid deadlocks (I'll discuss this later)
  • It is possible to block while holding a Symbian OS mutex. Symbian OS mutexes provide priority inheritance.

The freedom from the restrictions of the nanokernel mutex comes at a price in terms of performance; operations on Symbian OS mutexes are more complicated and hence slower than those on NFastMutex.

Our motivation in designing DMutex was the requirement that the mutex should be held, whenever possible, by the highest priority thread that requires the mutex. Of course, this is not possible if the mutex is already held when a higher-priority thread requests the mutex - in this case, the delay before the higher-priority thread acquires the mutex should be kept to a minimum. The design meets these criteria in these ways:

  1. The kernel defers a thread's acquisition of a mutex to the last possible moment. A thread cannot acquire a mutex on behalf of another thread; it can only acquire a mutex for itself. When a thread signals a mutex, the kernel does not directly hand the mutex over to the highest-priority waiting thread; it merely frees the mutex and releases the highest-priority waiting thread. The waiting thread must actually run to claim the mutex. We chose this design to take care of the case in which a high-priority thread acquires and releases a mutex several times in succession. Imagine we have a high-priority thread HAIKU that owns a mutex and a lower-priority thread EPIC that is the highest-priority thread waiting on that mutex. If HAIKU released the mutex and the kernel then handed it over to EPIC immediately, then HAIKU would not be able to reclaim the mutex so would be delayed by the lower-priority thread
  2. The kernel queues threads waiting for a mutex and releases them in priority order. So when the kernel releases a mutex the highest-priority waiting thread will run and acquire the mutex, if its priority is higher than the current thread's priority
  3. Mutexes implement priority inheritance. If a low priority thread, EPIC, is holding a mutex when a higher-priority thread, HAIKU, waits on the same mutex, then the kernel elevates the priority of EPIC to that of HAIKU. This ensures that another thread SONNET, of medium priority, which might prevent EPIC from running and releasing the mutex, does not delay HAIKU
  4. If the kernel suspends a thread that is waiting on a mutex, it removes it from the wait queue and places it on a separate suspended queue. The kernel no longer regards the thread as waiting for the mutex: the kernel will never hand over the mutex to a thread that is suspended. This is because this would result in the mutex being held for an indeterminate period.

We need three thread M-states to describe the interaction of a thread with a mutex. The most obvious of these is the EWaitMutex state, which indicates that the thread is blocked waiting for the mutex. The kernel adds a thread in EWaitMutex state to the priority-ordered mutex wait queue.

Point 4 requires the existence of a second state, EWaitMutexSuspended; this is because threads that are both waiting for a mutex and explicitly suspended are not enqueued on the wait queue but on the suspended queue. The kernel needs to perform different actions in the event, for example, of the thread being killed, and so different states are required.

Point 1 requires the existence of a third state, EHoldMutexPending. When the kernel releases a mutex, it marks the mutex as free and releases the highest-priority waiting thread. Typically, that thread will then run and acquire the mutex. Note that although there may be other threads on the wait queue, we release only one thread, the highest priority one. We do this for efficiency - there is no point releasing all those other threads since this is time-consuming and only the highest-priority thread among them will be able to acquire the mutex anyway. But this presents us with a problem: if we release the highest-priority thread and it is killed before it has had chance to acquire the mutex, then the mutex remains free but none of the waiting threads can claim it - because they are all blocked.

The solution is simply that the kernel moves the next highest-priority thread from the EWaitMutex state to EHoldMutexPending.

The thread that holds the mutex is in the EReady state. It is linked to the mutex by way of its cleanup queue and each mutex has a TThreadCleanup object embedded in it. When a thread acquires the mutex, the kernel adds the TThreadCleanup object to the thread's cleanup queue. (While the mutex remains free, the cleanup item is not linked to any queue and its cleanup item's thread pointer is NULL.) The priority of the cleanup item is equal to that of the highest-priority thread on the mutex wait queue. The kernel first sets this priority when a thread acquires the mutex, just before adding the cleanup item to the thread's cleanup queue. If another thread subsequently waits for the mutex then the kernel adjusts the priority of the cleanup item (if the new waiting thread has a higher priority than any other). Raising the cleanup item's priority may then cause the holding thread's priority to increase (if the new cleanup priority is higher than the holding thread's default priority). In this way, the holding thread's priority will never be less than that of any waiting thread, and so we get priority inheritance.

What happens if the holding thread is itself blocked on another mutex? In this case the kernel may elevate the priority of the holding thread of the second mutex if necessary. In principle, a sequence like this may be arbitrarily long, which would result in an arbitrarily long execution time for the mutex wait operation. To avoid this, we terminate the chain once we have traversed 10 mutexes. Priority inheritance will no longer operate if a chain of mutexes exceeds this length.

To ensure that the highest-priority thread always acquires the mutex first, the kernel must take action whenever a thread performs any of the following operations on a thread associated with the mutex:

  • Suspends
  • Resumes
  • Kills
  • Changes priority.

The following table summarizes the required actions. (The row indicates the relationship of the thread to the mutex, the column indicates the operation performed on the thread.)

Suspend Resume Kill Priority change
Holding thread No action. No action. Signal mutex. Priority of thread will not drop below that of highest-priority waiting thread.
Waiting Change thread state to wait/suspend. Adjust cleanup priority. Not applicable. Remove from wait queue and adjust cleanup priority. If priority raised and mutex free, make thread pending. If mutex not free adjust cleanup priority.
Waiting/suspended No action. If mutex free make thread pending else make thread waiting and adjust cleanup priority. Remove from suspended queue and adjust cleanup priority. No action.
Pending If mutex free and wait queue non-empty, make highest-priority waiting thread
No action. Remove thread from pending queue. If mutex
free and wait queue non-empty, make highest priority waiting thread pending.
If mutex free and thread priority
reduced below that of highest-priority
waiting thread, make the latter pending.

The kernel and memory model use DMutex extensively to protect global data structures that are accessed by multiple threads. The kernel protects DMutex operations using the system lock fast mutex.

Note that DMutex operations rely on fields that are present in DThread but not in NThread. Hence only Symbian OS threads may invoke mutex operations; it is not permitted for an IDFC or a non-Symbian OS thread to use a Symbian OS mutex.

We represent a Symbian OS mutex by the DMutex class. Like many others, this class is derived from DObject, which makes it a dynamically allocated reference counted object. DMutex is the kernel object referred to by a user-side RMutex handle:

class DMutex : public DObject
TInt Create(DObject* aOwner, const TDesC* aName,
TBool aVisible, TUint aOrder);
TInt HighestWaitingPriority();
void WaitCancel(DThread* aThread);
void WaitCancelSuspended(DThread* aThread);
void SuspendWaitingThread(DThread* aThread);
void ResumeWaitingThread(DThread* aThread);
void ChangeWaitingThreadPriority(DThread* aThread,
TInt aNewPriority);
void SuspendPendingThread(DThread* aThread);
void RemovePendingThread(DThread* aThread);
void ChangePendingThreadPriority(DThread* aThread,
TInt aNewPriority);
void WakeUpNextThread();
TInt Wait();
void Signal();
void Reset();
TInt iHoldCount;
TInt iWaitCount;
TUint8 iResetting;
TUint8 iOrder;
TUint8 iPad1;
TUint8 iPad2;
TThreadMutexCleanup iCleanup;
SDblQue iSuspendedQ;
SDblQue iPendingQ;
TThreadWaitList iWaitQ;
#ifdef _DEBUG
SDblQueLink iORderLink;
friend class Monitor;

Key member data of DMutex

Field Description
iHoldCount Count of the number of times the holding thread has waited on this mutex in a nested fashion. We increment this field if the holding thread waits again and decrement it if it signals the mutex; in the latter case we release the mutex if iHoldCount becomes zero.
iWaitCount Count of the number of waiting threads plus the number of waiting and suspended threads. This is used only to implement the RMutex::Count() method.
iResetting This flag is set while the mutex is being reset; this occurs just prior to the mutex being deleted and involves releasing and unlinking any waiting, suspended or pending threads. The flag is used to prevent further threads waiting on the mutex.
iCleanup A TThreadCleanup entry used both to enable the mutex to be released if the holding thread exits and also to enable the holding thread to inherit the priority of the highest-priority waiting thread. The iThread member of iCleanup is a pointer to the holding thread; a NULL value for iCleanup.iThread indicates that the mutex is free.
iSuspendedQ A doubly linked list of threads that are both waiting on the mutex and explicitly suspended. These threads have iWaitObj pointing to this mutex and have M-state EWaitMutexSuspended.
iPendingQ A doubly linked list of threads, released by the kernel after waiting on a mutex, but which have not yet claimed that mutex. These threads have iWaitObj pointing to this mutex and have M-state EHoldMutex Pending.
iWaitQ A 64-priority list of threads that are waiting on the mutex and that are not explicitly suspended. These threads have iWaitObj pointing to the mutex and have M-state EWaitMutex.

Condition variables - DCondVar

Often, a thread will need to block until some data that it shares with other threads reaches a particular value. In Symbian OS, it can do this by using a POSIX-style condition variable.

Condition variables provide a different type of synchronization to locking mechanisms like mutexes. Mutexes are used to make other threads wait while the thread holding the mutex executes code in a critical section. In contrast, a thread uses a condition variable to make itself wait until an expression involving shared data attains a particular state.

The kernel-side object is DCondVar, and its class definition is shown in the first of the two code samples which follow. Access to DCondVar from user-side is via RCondVar, which is shown in the second sample.

Condition variables are always used in association with a mutex to protect the shared data. In Symbian OS, this is, of course, an RMutex object.


class DCondVar : public DObject
TInt Create(DObject* aOwner, const TDesC* aName, TBool aVisible);
void WaitCancel(DThread* aThread);
void WaitCancelSuspended(DThread* aThread);
void SuspendWaitingThread(DThread* aThread);
void ResumeWaitingThread(DThread* aThread);
void ChangeWaitingThreadPriority(DThread* aThread, TInt aNewPriority);
TInt Wait(DMutex* aMutex, TInt aTimeout);
void Signal();
void Broadcast(DMutex* aMutex);
void Reset();
void UnBlockThread(DThread* aThread, TBool aUnlock);
TUint8 iResetting;
TUint8 iPad1;
TUint8 iPad2;
TUint8 iPad3;
DMutex* iMutex;
TInt iWaitCount;
SDblQue iSuspendedQ;
TThreadWaitList iWaitQ;
friend class Monitor;


class RCondVar : public RHandleBase
IMPORT_C TInt CreateLocal(TOwnerType aType=EOwnerProcess);
IMPORT_C TInt CreateGlobal(const TDesC& aName,
TOwnerType aType=EOwnerProcess);
IMPORT_C TInt OpenGlobal(const TDesC& aName,
TOwnerType aType=EOwnerProcess);
IMPORT_C TInt Open(RMessagePtr2 aMessage, TInt aParam,
TOwnerType aType=EOwnerProcess);
IMPORT_C TInt Open(TInt aArgumentIndex,
TOwnerType aType=EOwnerProcess);
IMPORT_C TInt Wait(RMutex& aMutex);
IMPORT_C TInt TimedWait(RMutex& aMutex, TInt aTimeout);
IMPORT_C void Signal();
IMPORT_C void Broadcast();

You can see that the condition to be tested lives outside of the condition variable. This is because the condition is application-defined, and allows it to be as complex as you wish. The only real requirement is that the volatile data being tested by the condition is protected by the mutex that is being used with the condition variable.

Let's look at how condition variables are used in practice.

One common use case for condition variables is the implementation of thread-safe message queues, providing a producer/consumer communication mechanism for passing messages between multiple threads. Here we want to block producer threads when the message queue is full and we want to block consumer threads when the queue is empty. Assume that the queue is a doubly linked list of messages. Clearly we will need to protect this list with a mutex (let's call it myMutex), so that producers and consumers do not interfere with each other as they add and remove messages from the queue.

Now let's look at this from the point of view of the consumer thread, and let's suppose that there are no messages for it to process. How does it deal with this situation? The consumer thread could repeatedly lock and unlock myMutex, each time checking the linked list for more messages. But this busy polling is extremely inefficient. It is far better for the consumer thread to first wait on myMutex, and then on the condition variable, by calling RCondVar::Wait(RMutex&aMutex). This method will only return when there is a new message on the list. What goes on behind the scenes when this happens?

The consumer thread ends up in DCondVar::Wait(). This is kernel code, but it is running in the consumer thread's context. Almost the first thing DCondVar::Wait() does is to release the mutex. The function then blocks - this means the consumer thread does not run again until the condition variable is signaled. Now that myMutex is unlocked, other threads that were blocked on the mutex may become runnable, and can access and modify the message-status variable.

Suppose a producer thread signals the condition variable. What then happens to our sleeping consumer thread? Immediately after it wakes, still in DCondVar::Wait(), it re-aquires the mutex, preventing access by other threads and making it safe for it to examine the shared data itself.

So we have seen that usage pattern for condition variables is as follows:

// possible race condition here if signalling thread
// does not hold mutex

Here CONDITION is an arbitrary condition involving any number of user-side variables whose integrity is protected by the mutex. You have to loop while testing the condition since there is no guarantee that the condition has been satisfied when the condition variable is signaled. Different threads may be waiting on different conditions or the condition may have already been absorbed by another thread. All that can be said is that the thread will awaken whenever something happens which might affect the condition.

And what about the producer thread? How does it signal the condition variable? There are two methods in RCondVarthat allow it to do this:

Signal() and Broadcast().

Signal() unblocks a single, waiting thread. If there are several of these, then the kernel unblocks the highest-priority thread that is not explicitly suspended. If there are no threads currently waiting this call does nothing.

The calling thread does not have to hold the mutex when it calls Signal() but we recommend that it does so. Otherwise a race condition can result if it signals the condition variable just between the waiting thread testing the condition and calling Wait().

Broadcast() unblocks all the threads that are waiting on the condition variable. As for Signal(), it is best if the thread that calls Broadcast() holds the mutex first.

I hope that I have also shown that, although RCondVar is used for explicit communications between threads, the communications are anonymous. The producer thread does not necessarily know that the consumer thread is waiting on the condition variable that it signaled. And the consumer thread does not know that it was the producer thread that woke it up from its wait on the condition variable.

Symbian OS thread death

All Symbian OS threads have an exit handler installed, which performs first stage cleanup of the thread in the context of the exiting thread. This first stage includes:

  • Restoring the thread to a consistent state - for example removing the thread from any mutexes or semaphores on which it was waiting
  • Running thread-exit cleanup handlers
  • Completing logons.

The kernel can't perform its entire cleanup in the exiting thread's context - for example it can't free the thread's supervisor stack or control block. So the kernel performs a second phase of exit processing in the supervisor thread context. Each DThread has a DFC, given by iKillDfc, which is queued on the supervisor thread just before the exiting thread actually terminates. The DFC completes the cleanup process - closing thread handles, freeing stacks and freeing the thread control block. If this is the last thread in a process, then it also closes process handles and frees the process control block.

Types of Symbian OS thread

When we start to think about thread death, there are four different types of Symbian OS thread we need to consider. These are specified by the iFlags member data, and defined in u32std.h:

KThreadFlagProcessCritical = 1
KThreadFlagProcessPermanent = 2
KthreadFlagSystemCritical = 4
KthreadFlagSystemPermanent = 8

The corresponding user-side enumerations are in e32std.h:

enum TCritical

The meanings of the values in this enumeration are:

Value Descriotion
ENotCritical The thread or process is not critical. No special action is taken on thread or process exit or panic.
EProcessCritical Indicates that a thread panic causes the process to panic.
EProcessPermanent Indicates that a thread exit of any kind causes the process to exit.
EAllThreadsCritical Indicates that if any thread in a process panics, then the process panics.
ESystemCritical Indicates that a thread or process is system-critical. If that thread or process panics, then the entire system is rebooted. Clearly this is a drastic step to take, so we ensure that only a process with the Protected Server capability can set a thread to system-critical.
ESystemPermanent Indicates that a thread or process is system-permanent. If that thread or process exits, then the entire system is rebooted. Clearly this too is a drastic step to take, so again we ensure that only a process with the Protected Server capability can set a thread to system-permanent.

Kernel threads

The Symbian OS kernel itself creates five kernel threads at boot time, and these threads continue to run until the mobile phone is rebooted. (In fact, there may even be more than five threads on a phone, since kernel extensions can create them too.) Next I will briefly describe each kernel thread and its purpose in the system.

The null thread

Earlier I described how the null thread (also known as the idle thread) is the first thread to run on a device at boot time. This thread's execution begins at the reset vector. Just after the reset is applied, there are no NThread or DThread objects in existence, of course - but the thread of execution that begins here eventually becomes the null thread, which is the thread with the lowest possible priority in the system. Since there is no point in time slicing the null thread, the kernel sets a variable in the thread to disable time slicing. This thread has iType==EThreadInitial, and is the only thread in the system with this type.

Because of the unusual way this thread comes into being, the bootstrap must allocate and map its stack before kernel execution begins. This means that the stack is in a special place - at the start of the chunk containing the kernel heap. For more on Symbian OS bootup, see Chapter 16, Boot Processes.

As we said, the null thread has the lowest possible priority in the system. This means that the null thread will gain control only when no other thread is ready to run. Generally the null thread simply loops forever executing a wait for interrupt instruction, which places the CPU in a low-power mode where instruction execution stops until a hardware interrupt is asserted.

The main task that the null thread performs in its infinite loop is to delay the nanokernel timer tick for as long as possible. The null thread inspects the nanokernel timer queue and determines how many ticks will elapse before the first timer is due to expire. It then updates the hardware timer to skip that number of ticks, so that the next timer interrupt coincides with the expiry of the first timer on the queue. In this way, we save power by not reactivating the CPU several times just to discover that there is nothing to be done. For more details of Symbian OS power management, see Chapter 15, Power Management.

The supervisor thread

This is the second thread to run after a system reset. It is responsible for the final phase of kernel initialization and for phase 3 of the variant initialization, which initializes the interrupt dispatcher and enables the nanokernel tick timer interrupt. For more details of the supervisor thread's role in start-up, see Chapter 16, Boot Processes.

Once the OS is running, the primary functions of the supervisor thread are cleanup activities and providing notification of non-time-critical events to user-side code. The supervisor thread's priority is set so that it is higher than applications and most user-mode code but lower than anything that is time-critical.

The supervisor thread performs many of its functions via deferred function calls, or DFCs, which I will discuss in Chapter 5, Interrupts and Exceptions. DFCs that run in the supervisor thread can't expect any real-time guarantees - and would typically be unbounded operations anyway, often involving the freeing of memory. To sum up, DFCs in the supervisor thread perform these tasks:

  1. Thread and process cleanup on exit
  2. Asynchronous deletion. For more on this, see Chapter 7, Memory Models
  3. Asynchronous change notifier completion. (User code uses the RChangeNotifier class to get notification from the kernel of important changes, such of change of locale.) Sometimes the kernel needs to signal change notifiers either from time-critical code (for example the midnight crossover detection in TSecondQ) or from places where low-level kernel mutexes are held (for example the memory threshold detection code may run with the kernel heap mutex held). The signaling of change notifiers is unbounded (since there may be arbitrarily many of them) and involves waiting on the change notifier container's mutex. To avoid timing and deadlock problems, the kernel provides the function Kern::AsyncNotify Changes(). This function accumulates the set of changes that it needs to signal in a bit mask, K::AsyncChanges, and queues a DFC on the supervisor thread to signal the change notifiers
  4. Asynchronous freeing of physical RAM. For more on this, see Chapter 7, Memory Models
  5. The completion of publish and subscribe property subscriptions.

DFC thread 0

This thread has priority 27 (usually the second highest in the system), and it simply runs a DFC queue. The kernel does not use this queue itself, but provides it for those device drivers that do not have stringent real-time requirements, to give them a context that they can use for their non-ISR processing. You can think of this queue as approximately equivalent to the single DFC queue that EKA1 provided. The presence of this single general purpose DFC thread in the kernel saves memory, because each device driver does not have to create its own thread. The function Kern::DfcQue0() returns a pointer to the DFC queue serviced by this thread.

Symbian device drivers that use this thread include serial comms, sound, ethernet, keyboard and digitizer.

DFC thread 1

This thread has priority 48; it is generally the highest-priority thread in the system. The nanokernel timer DFC runs on this thread.

DFC thread 1 is available for use by other device drivers if necessary; the function Kern::DfcQue1()returns a pointer to the DFC queue serviced by this thread.

You should beware of using DFC thread 1 for anything other than running the nanokernel timer queue. You should also beware of creating a higher-priority thread than this one. If you delay the nanokernel timer DFC by more than 16 nanokernel timer ticks, you run the risk of adversely affecting the accuracy of the nanokernel timers.

The timer thread

This thread has priority 27. It manages the Symbian OS timer queues. The timer thread is also available for device drivers to use if they need it - the function Kern::TimerDfcQ()returns a pointer to the DFC queue serviced by this thread.

What is a process?

Under Symbian OS, a process is both a single instantiation of an executable image file and a collection of one or more threads that share a particular address space (or memory mapping). This address space may, or may not, be different to that of other processes in the system - this depends on whether the processor in the mobile phone has an MMU, and on which particular memory model the phone manufacturer has chosen to use. In most cases, designers will want to ensure that they protect processes from each other by choosing the memory model appropriately. In this case, a thread in one process will not be able to directly access the memory belonging to a thread in another process - although, it will of course be able to directly access the memory of any thread in the same process. So you can see that, with the appropriate choice of memory model, the process is the fundamental unit of memory protection under Symbian OS.

The loader creates a process by first asking the kernel to create a DProcess object, then loading the image and informing the kernel that it has done so. The kernel creates a single thread, marks it as the main thread, and starts execution at the process's entry point. The main thread is marked as KThreadFlagProcessPermanent(see Section 3.3.8), but the application can change this later. As well as sharing an address space, the threads in a process are connected in other ways:

  • You can specify their priorities relative to the process priority; changing the process priority will change the priorities of all such threads. (You may also specify absolute thread priorities - these do not change when the process priority is changed)
  • If the process exits or is terminated, then the kernel terminates all the threads in the process with the same exit information
  • Threads in the same process can share object handles; this is not possible for threads in different processes
  • A user thread can create a new thread only in its own process.

This section is quite a lot shorter than the one on threads, since the two things that are mainly of interest for processes are how they are loaded, which I will cover in Chapter 10, The Loader, and the role they play in address spaces, which I will cover in Chapter 7, Memory Models. All that remains is to discuss the Symbian OS representation of the process, DProcess.

DProcess class

Like many other classes in the kernel, this class is derived from DObject, which makes it a dynamically allocated reference counted object. DProcess is the kernel object referred to by a user-side RProcess handle.

Here a cut-down version of the class:

class DProcess : public DObject
TInt Create(TBool aKernelProcess, TProcessCreateInfo& aInfo,HBuf* aCommandLine);
TInt SetPriority(TProcessPriority aPriority);
TInt Logon(TRequestStatus* aStatus, TBool aRendezvous);
void Rendezvous(TInt aReason);
TInt AddCodeSeg(DCodeSeg* aSeg, DLibrary* aLib, SDblQue& aQ);
TInt RemoveCodeSeg(DCodeSeg* aCodeSeg, SDblQue* aQ);
TBool HasCapabilityNoDiagnostic(TCapability aCapability);
virtual TInt NewChunk(DChunk*& aChunk, SChunkCreateInfo& aInfo,TLinAddr& aRunAddr)=0;
virtual TInt AddChunk(DChunk* aChunk,TBool isReadOnly)=0;
virtual TInt DoCreate(TBool aKernelProcess,TProcessCreateInfo& aInfo)=0;
virtual TInt Loaded(TProcessCreateInfo& aInfo);
TInt NewThread(DThread*& aThread, SThreadCreateInfo& aInfo,TInt* aHandle, TOwnerType aType);
virtual void Resume();
void Die(TExitType aType,TInt aReason,const TDesC &aCategory);
TInt iPriority;
SDblQue iThreadQ;
TUint8 iExitType;
TUint8 iPad1;
TUint8 iPad2;
TUint8 iPad3;
TInt iExitReason;
TBufC<KMaxExitCategoryName> iExitCategory;
DObjectIx* iHandles;
TUidType iUids;
TInt iGeneration;
TUint iId;
TUint32 iFlags;
HBuf* iCommandLine;
DProcess* iOwningProcess;
SDblQue iTargetLogons;
RArray<SCodeSegEntry> iDynamicCode;
SSecurityInfo iS;
SSecurityInfo iCreatorInfo;
TUint iCreatorId;
TUint iSecurityZone;
TInt iEnvironmentData[KArgIndex];
enum TProcessAttributes
TInt iAttributes;
TLinAddr iDataBssRunAddress;
DChunk* iDataBssStackChunk;
DCodeSeg* iCodeSeg;
DCodeSeg* iTempCodeSeg;
DMutex* iProcessLock;
DMutex* iDllLock; // can be held while in user mode
// user address to jump to for new threads, exceptions
TLinAddr iReentryPoint;

Key member data of DProcess

Field Description
iThreadQ Doubly linked list of all threads belonging to this process. Accesses to this list are protected by the process lock mutex.
iHandles Pointer to array (DObjectIx) of process-global handles for this process.
iDataBssStackChunk Pointer to chunk-holding process global data (.data and .bss sections) and, in most cases, user stacks of threads belonging to this process. The memory model determines whether or not user stacks are placed in this chunk.
iDataBssRunAddress Run address of base of initialized data.
iDynamicCode Array listing all explicitly dynamically loaded code segments which are attached to this process - that is, only ones corresponding to DLLs which have been explicitly loaded, not the process EXE code segment (iCodeSeg) or code segments which are attached to the process only due to implicit linkages from other code segments. For each such code segment this array contains two pointers - a pointer to the DCodeSeg object and a pointer to the DLibrary or (for the kernel process only) the DLogicalDevice DPhysicalDevice object that represents the process's use of the code segment.
iCodeSeg Pointer to DCodeSeg object that represents the executable image used to create this process. This value is set when the process is fully loaded and ready to be resumed.
iTempCodeSeg Temporary pointer to DCodeSeg object that represents the executable image used to create this process. This value is only used during process creation; it is set to NULL when the process is fully loaded and ready to be resumed.
iAttributes Process attributes. Some of these are generic (private, supervisor, being loaded, resumed); the memory model defines some more.
iFlags Process flags (just-in-time debug).
iProcessLock Pointer to DMutex object used to protect the process thread list and in some memory models, the process address space list.
iDllLock Pointer to DMutex object used to protect DLL static data constructors and destructors that run user-side in this process. Note that this mutex is held while running user-mode code and hence the thread need not enter a critical section before acquiring it; these are the only mutexes used by the kernel with this property.

Processes in the emulator

The increased code-sharing in the EKA2 emulator means that the EKA2 emulator provides much better emulation of Symbian OS processes than the EKA1 emulator did. The emulator can instantiate a process from a .EXE file and has the same object and thread ownership model as on target hardware. The EKA1 emulator failed to emulate processes at all.

That said, debugging multiple processes on a host OS is difficult and so the EKA2 emulator still executes as a single process in the host OS. Indeed, the emulator does not provide a complete emulation of processes as found on target platforms. In particular, the following aspects of processes are not completely emulated:

  1. Memory protection between processes, or between user and kernel mode
  2. Full support for multiple instances of a DLL with writable static data.

The EKA1 emulator shares both these failings.

When a .EXE is not an EXE

Under Windows, a process is instantiated from a Portable Executable (PE) format file of type EXE. However, a process instantiated in this way may not load and link further EXE files. This means that the emulator must use a PE file of type DLL to create a new emulated Symbian OS process.

The Symbian tool chain could build all EXE targets for the emulator as host DLLs. These would then be easily loaded into the emulator as an emulated process. Unfortunately this prevents the EXE from being invoked from the Windows command line, something which has proved useful with EKA1. It also makes it impossible to load multiple instances of a process that has static data, because static data support is provided by the host OS, and the emulator is a single process in the host OS.

We solved this dilemma by making certain assumptions about the PE file format and the Windows platform:

  1. The difference between a DLL and an EXE is a single bit in the PE file header
  2. Identical copies of a DLL with different file names load as independent modules in the host OS.

This is currently true for all the Win32 platforms we have tested. When creating a process within the emulator, we go through the following steps:

  1. We copy the EXE file to a new filename
  2. We set the DLL type bit and clear the Win32 entry point
  3. We load the copy as a DLL
  4. We find the Symbian OS entry point by looking for the _E32Startup export.

As a result, if you set the target type to EXE, then the tool chain creates a file that can bootstrap the emulator and that can also be loaded as a process within the emulator multiple times.

Entry points

The DLL and EXE entry point scheme used in the emulator has been changed, allowing the emulator to fully control how and when these functions are called. It also enables the Symbian OS EXE scheme I have just described.

The entry point in a Win32 DLL is not used at all, and the entry point for a Win32 EXE merely bootstraps the emulator by calling BootEpoc().

The Symbian OS entry points are implemented by exporting a named symbol from the DLL or EXE. The DLL entry point is _E32Dll and the EXE entry point is _E32Startup . The export management tools recognize these symbols and ensure that they are not included in frozen exports files and are always exported by name as the last symbol from the file.


EKA2 implements a priority-driven, preemptive scheduling scheme. The highest-priority ready thread, or one of several ready threads of equal highest priority, will usually run. (The exception, which I discussed earlier, is when a high-priority thread is waiting on a nanokernel fast mutex held by a lower-priority thread - in this case, the lower-priority thread runs.)

Within Symbian OS, scheduling is the responsibility of the nanokernel. Threads that are eligible for execution (that is, threads that are not waiting for an event) are called ready and are kept on a priority-ordered list, the ready list.

Each nanothread has an integer priority between 0 and 63 inclusive. As we have said, the highest-priority thread that is ready to run will run. If there are several threads at the same priority, then one of two things may happen. Firstly, the threads may be executed in a round-robin fashion, with the timeslice selectable on a thread-by-thread basis. Secondly, they may be executed in FIFO order - that is the first thread to become ready at that priority will run until it blocks and other threads at the same priority will not run until the first thread blocks. Which of these two methods is chosen is a property of the thread itself, not the scheduler.

Each thread has its own timeslice (iTimeslice) and time count (iTime). Whenever the thread blocks or the kernel rotates the thread to the end of the queue of threads at the same priority, the kernel sets the iTime field equal to iTimeslice. The low-level tick interrupt then decrements the current thread's iTime if it is positive and triggers a reschedule if it becomes zero. So you can see that if iTimeslice is positive, the thread will run for iTimeslice low level timer ticks before yielding to the next thread at the same priority. If iTimesliceis negative, the thread will only yield to other threads at the same priority if it blocks.

The ready list, shown graphically in Figure 3.2, holds all the threads that are currently eligible for execution. It is always accessed with the kernel locked so, to maintain low thread latency, operations on the ready list need to be bounded and as fast as possible. We achieve this by using 64 separate queues, one for each possible thread priority - the kernel places each ready thread in the queue corresponding to its priority.

Figure 3.2 The ready list

The kernel also maintains a 64-bit mask to indicate which queues have entries; bit n in the mask is set if and only if the queue for priority n has entries.

So, to insert an entry, the kernel simply adds it to the tail of the queue corresponding to its priority (no searching is required) and sets the corresponding bit in the bit mask. To remove an entry, the kernel first unlinks it from its queue, then, if that queue is now empty, resets the bit in the bit mask. To find the highest-priority entry, the kernel finds the most significant 1 in the bit mask (which can be done with a binary search or with a single instruction on some CPUs), and then finds the first entry on the corresponding queue. You can see that this implementation gives us bounded (and small) execution times for insertion and removal of entries and for finding the highest-priority entry.

To save on memory, we use a single pointer for each queue. This is NULL if the queue is empty, otherwise it points to the first entry on the queue. We arrange the entries in the queue in a doubly linked ring. We use the same priority ordered list implementation for DFC queues, and semaphore and mutex wait queues.

The kernel also maintains a flag (TheScheduler.iReschedule-NeededFlag) that indicates whether a thread switch may be required. It sets this flag whenever it adds a thread to the ready list, and that thread's priority is greater than the highest priority of any other thread already on the list. The nanokernel timer tick interrupt also sets this flag when the current thread's timeslice has expired.

When the kernel subsequently becomes unlocked, it checks this flag to determine whether a reschedule is needed, and clears the flag when it performs the reschedule.

The scheduler is responsible both for running IDFCs (immediate deferred function calls) and for selecting the next thread to run and switching to it. It does not do any page table manipulations to switch processes - instead it provides a hook that the Symbian OS memory model uses to arrange for such code to be called.

Here is a summary of the scheduler control block:

Field Description
iPresent[2] 64-bit mask with a 1 in bit position n if and only if iQueue[n] is non-empty, that is, if and only if there is a thread of priority n on the ready list.
iRescheduleNeededFlag Boolean flag that is set if a reschedule is needed when the kernel is locked, and cleared by the scheduler when it runs.
iDfcPendingFlag Boolean flag, set when an IDFC is queued and cleared
iKernCSLocked The kernel lock (otherwise known as the preemption lock).
iDfcs Doubly linked list of pending IDFCs.
iMonitorExceptionHandler Pointer to exception handler installed by the crash debugger.
iProcessHandler Pointer to the process address space switch handler in the Symbian OS memory model.
iLock The system lock fast mutex.
iCurrentThread Pointer to the currently executing NThread.
iAddressSpace The identifier of the currently active process address space.
iExtras[16] Space reserved for extra data used by the Symbian OS process switch handler.

Now let's look at the full scheduler algorithm. Before we start, we'll define some terms.

Non-volatile registers: those CPU registers that are preserved across a function call (r4--r11 and r13 on ARM).

CPU-specific registers: those registers, outside the normal supervisor-mode register set, that are required to exist on a per-thread basis. Examples on ARM include the DACR (domain access control register, CP15 CR3) and user mode R13 and R14.

System lock: a fast mutex, iLock, which the kernel uses to protect address space changes, among other things.

Kernel lock: a counter, iKernCSLocked, which is always = 0. Zero is the normal state and indicates that IDFCs and a reschedule may run immediately following an interrupt. Non-zero values indicate that IDFCs and reschedules must be deferred until the count is decremented to zero. Also known as the preemption lock.

The kernel calls the scheduler in two different situations. The first is at the end of an interrupt service routine, if the following conditions are met:

  1. IDFCs are pending or a reschedule is pending
  2. The kernel is not locked (iKernCSLocked==0)
  3. On ARM the interrupted mode must be usr or svc. (Other modes are non-preemptible since they don't have per-thread stacks.)

The kernel also calls the scheduler when it becomes unlocked (iKernCSLocked decrements to zero), if the following condition is met:

4. Either IDFCs are pending or a reschedule is pending. (This will be the case if the current thread has just completed a nanokernel function that has blocked it or made a higher-priority thread ready.)

Here is the full scheduler algorithm for the moving memory model:

// Enter with kernel locked
// Active stack is supervisor stack of current thread
1 Disable interrupts
2 start_resched:
3 IF IDFCs pending (iDfcPendingFlag TRUE)
4 Run IDFCs (with interrupts enabled but kernel locked)
5 iDfcPendingFlag = FALSE
7 IF reschedule not needed (iRescheduleNeededFlag FALSE)
8 iKernCSLocked=0 (unlock the kernel)
9 Return
11 Reenable interrupts
12 Save nonvolatile registers and CPU specific registers on stack
13 iCurrentThread->iSavedSP = current stack pointer
14 iRescheduleNeededFlag = FALSE
15 next_thread = first thread in highest priority non-empty ready queue
16 IF next_thread->iTime==0 (ie timeslice expired)
17 IF another thread is ready at same priority as next_thread
18 IF next_thread holds a fast mutex
19 next_thread->iHeldFastMutex->iWaiting=TRUE
20 goto resched_end
22 next_thread->iTime=next_thread->iTimeslice (new timeslice)
23 next_thread=thread after next_thread in round-robin order
26 IF next_thread holds a fast mutex
27 IF next_thread holds system lock
28 goto resched_end
30 IF next_thread requires implicit system lock
31 IF system lock is held OR next_thread requires address space switch
32 next_thread->iHeldFastMutex->iWaiting=TRUE
35 goto resched_end
37 IF next_thread is blocked on a fast mutex
38 IF next_thread->iWaitFastMutex->iHoldingThread (mutex is still locked)
39 next_thread=next_thread->iWaitFastMutex->iHoldingThread
40 goto resched_end
43 IF next_thread does not require implicit system lock
44 goto resched_end
46 IF system lock held by another thread
47 next_thread=system lock holding thread
48 system lock iWaiting = TRUE
49 goto resched_end
51 IF thread does not require address space switch
52 goto resched_end
54 iCurrentThread=next_thread
55 current stack pointer = next_thread->iSavedSP
56 Restore CPU specific registers from stack
57 system lock iHoldingThread = next_thread
58 next_thread->iHeldFastMutex = system lock
59 Unlock kernel (scheduler may go recursive here, but only once)
60 Invoke address space switch handler
61 Lock the kernel
62 system lock iHoldingThread = NULL
63 next_thread->iHeldFastMutex = NULL
64 IF system lock iWaiting (there was contention for the system lock)
65 system lock iWaiting = FALSE
66 iRescheduleNeededFlag = TRUE (so we reschedule again)
67 IF next_thread has critical section operation pending
68 Do pending operation
71 goto switch_threads
72 resched_end: // switch threads without doing process switch
73 iCurrentThread=next_thread
74 current stack pointer = next_thread->iSavedSP
75 Restore CPU specific registers from stack
76 switch_threads:
77 Restore nonvolatile registers from stack
78 Disable interrupts
79 IF IDFCs pending or another reschedule needed
80 goto start_resched
82 iKernCSLocked=0 (unlock the kernel)
83 Return with interrupts disabled

Lines 1-10 are concerned with running IDFCs. We check for the presence of IDFCs with interrupts disabled, since an interrupt could add an IDFC. If IDFCs are present, we call them at this point; we do this with interrupts enabled and with the kernel locked. IDFCs run in the order in which they were added to the queue. An IDFC could add a thread to the ready list; if that thread has a sufficiently high priority, the kernel sets iRescheduleNeededFlag. We remove each IDFC from the pending queue just before we execute it. We then disable interrupts again and make another check to see if more IDFCs are pending. If there are no more IDFCs, the kernel clears the iDfcPendingFlag. Interrupts are still disabled here. Next we check iRescheduleNeededFlag. This could have been set in several different ways:

  1. By the current thread just before calling the scheduler (for example if that thread needs to block waiting for an event)
  2. By an IDFC
  3. By the timer tick interrupt, if the current thread's timeslice has expired.

If the flag is clear, no further action is needed and the scheduler returns. If it is set, we proceed to switch threads.

Lines 11-14 are straightforward: we can re-enable interrupts at this point since the most they could do is queue another IDFC that would eventually cause the scheduler to loop. We save the current thread's register context on the stack, which is the thread's own supervisor mode stack. Then we store the stack pointer in the current thread's iSavedSP field and clear iRescheduleNeededFlag since we are about to do a reschedule.

Line 15 implements the basic scheduling policy. The most significant bit in the 64-bit mask indicates the highest priority of any ready thread. We select the first thread on the queue corresponding to that priority as a candidate to run.

Lines 16-25 deal with round-robin scheduling of threads at the same priority. The system tick interrupt decrements the current thread's iTime field; when this reaches zero the thread's timeslice has expired, so the iRescheduleNeededFlag is set, which causes a reschedule at the next possible point - either at the end of the tick ISR or when the kernel is next unlocked. Line 16 checks to see if the selected thread's timeslice has expired. If it has, and there is another thread at the same priority, and the originally selected thread does not hold a fast mutex, then we select the next thread in round-robin order and we reset the original thread's timeslice.

If the original thread does hold a fast mutex, we defer the round-robin and set the fast mutex iWaiting flag so that the round-robin will be triggered when the thread releases the mutex. We defer the round-robin to reduce the time that might be wasted by context switching to another thread that then immediately waits on the same mutex and causes another context switch. This would be a particular problem with threads waiting on the system lock. We expect that a fast mutex will only be held for short periods at a time and so the overall pattern of round-robin scheduling will not be disturbed to any great extent.

Lines 26-36 deal with the case where the selected thread holds a fast mutex. If the thread holds the system lock, we can simply switch straight to the thread with no further checking, since the address space cannot have been changed since the thread last ran. Also, the thread cannot be blocked on another fast mutex (because it holds one and they do not nest).

If the selected thread holds a fast mutex other than the system lock, we still switch to it, and we don't have to call out to do address space changes, since we don't guarantee that the user-mode address space is valid during a critical section protected by a fast mutex (unless it's the system lock). However, if an address space change would normally be required, we set the mutex iWaiting flag to ensure that the address space change does actually occur when the fast mutex is released. In addition, if the thread has the KThreadAttImplicitSystemLock attribute and the system lock is currently held, we set the mutex iWaiting flag. This is to ensure that the thread doesn't exit the mutex-protected critical section while the system lock is held.

Lines 37-42 deal with the case where the selected thread is actually blocked on a fast mutex. Such threads stay on the ready list, so the kernel may select them during a reschedule. We do not want to waste time by switching to the thread and letting it run, only to immediately switch to the holding thread. So we check for this case in the scheduler and go straight to the mutex holding thread, thus saving a context switch. This check also guarantees that the YieldTo function used in NFastMutexwait operations cannot return until the mutex has been released. Notice that we need to check both iWaitFastMutex and iWaitFastMutex->iHoldingThread, since when the holding thread releases the mutex, iHoldingThread will be set to NULL but iWaitFastMutex will still point to the mutex. As before, there is no need to do any address space changing if we switch to the mutex holding thread. There is also no need to set the fast mutex iWaiting flag here since it must already have been set when the selected thread blocked on it.

Lines 43-50 deal with threads requiring an implicit system lock. We mainly use this mechanism for threads requiring long-running address space switches: to perform such a switch the scheduler must claim the system lock. Threads that do not need implicit system lock will also not need the scheduler to call the address-space-switch hook; the scheduler can simply switch to them at this point (lines 43-45). If the selected thread does require an implicit system lock, the scheduler then checks if the lock is free. If it is not, the scheduler switches to the system lock holding thread. It also sets the system lock's iWaiting flag since there is now a thread implicitly waiting on the system lock.

If we reach line 51, the selected thread needs an implicit system lock and the lock is free. If the thread does not need an address space change, we can now switch it in - the system lock is not claimed (lines 51-53). If the thread does require an address space change - that is, it has the KThreadAttAddressSpace attribute and its iAddressSpace value differs from the currently active one - then the scheduler calls out to do this address space change (line 60).

In lines 54-56 we do the actual thread switch. We change stacks and restore the CPU-specific registers from the new thread's stack. At this point, the new thread is effectively running with the kernel locked.

Lines 57-58 claim the system lock for the new thread - we know that it's free here. Then we unlock the kernel (line 59). From this point on, further preemption can occur. It's also worth noting that the scheduler will go recursive here if an IDFC was queued by an interrupt serviced during the first part of the reschedule. There can only be one recursion, however, since the second reschedule would find the system lock held and so could not reach the same point in the code.

Line 60 calls the address space switch handler in the Symbian OS memory model to perform any MMU page table manipulations required to change to the required address space. The switch handler is also responsible for changing the iAddressSpace field in the scheduler to reflect the new situation. Since the switch handler runs with the kernel unlocked, it does not directly affect thread latency. But it does affect latency indirectly, since most Symbian OS user mode threads need an address space switch to run and many kernel functions wait on the system lock. The switch handler also affects the predictability of execution time for Symbian OS kernel functions. This means that we want the system lock to only be held for a very short time. To accommodate this, the address space switch handler does the operation in stages and checks the system lock's iWaiting flag after each stage. If the flag is set, the switch handler simply returns and we trigger a further reschedule (line 64) to allow the higher-priority thread to run. We set the iAddressSpace field in the scheduler to NULL just before we make the first change to the address space and we set it to the value corresponding to the new thread just after the last change is made. This ensures that we take the correct action if the address space switch handler is preempted and another reschedule occurs in the middle of it.

After we've done the address space switch, we lock the kernel again and release the system lock (lines 61-70). If contention occurred for the lock, we do not call the scheduler again directly as would normally be the case in a fast mutex signal; instead we set the iRescheduleNeededFlag, which will cause the scheduler to loop. Attempting recursion at this point would be incorrect, because the system lock is now free and there would be nothing to limit the recursion depth. [The processing of a deferred critical section operation (line 68) could cause the scheduler to go recursive, but only in the case where the thread was exiting; clearly, this cannot occur more than once. A deferred suspension would simply remove the thread from the ready list and set the iRescheduleNeededFlag,which would then cause the scheduler to loop.]

Lines 73-75 cover the case in which an address space switch was not required. They do the actual thread switch by switching to the new thread's stack and restoring the CPU-specific registers.

In lines 76 and onwards we finish the reschedule. First we restore the non-volatile registers, then we disable interrupts and make a final check of iDfcPendingFlag and iRescheduleNeededFlag. We need to do this because interrupts have been enabled for most of the reschedule and these could either have queued an IDFC or expired the current thread's timeslice. Furthermore, the iRescheduleNeededFlag may have been set because of system lock contention during the processing of an address space change. In any case, if either of these flags is set, we loop right back to the beginning to run IDFCs and/or select a new thread. If neither of the flags is set, then we unlock the kernel and exit from the scheduler in the context of the new thread.

The scheduler always returns with interrupts disabled. We need to make sure of this to prevent unbounded stack usage because of repeated interrupts in the same thread. When an interrupt occurs, the kernel pushes the volatile registers (that is, those modified by a normal function call) onto the thread's supervisor stack before calling the scheduler. If interrupts were re-enabled between unlocking the kernel and popping these registers, another interrupt could occur and push the volatile registers again before causing another reschedule. And so it might continue, if interrupts were not disabled.

The algorithm and the explanation above apply to the moving memory model. The emulator and direct memory models do not do any address space switching, simplifying scheduling. The multiple memory model uses a simplified address space switching scheme, since switching is very fast. The multiple memory model scheduling algorithm becomes:

// Enter with kernel locked
// Active stack is supervisor stack of current thread
1 Disable interrupts
2 start_reschedule:
3 IF IDFCs pending (iDfcPendingFlag TRUE)
4 Run IDFCs (with interrupts enabled but kernel locked)
5 iDfcPendingFlag = FALSE
7 IF reschedule not needed (iRescheduleNeededFlag FALSE)
8 iKernCSLocked=0 (unlock the kernel)
9 Return
11 Reenable interrupts
12 Save non-volatile registers and CPU specific registers on stack
13 iCurrentThread->iSavedSP = current stack pointer
14 iRescheduleNeededFlag = FALSE
15 next_thread = first thread in highest priority non-empty ready queue
16 IF next_thread->iTime==0 (ie timeslice expired)
17 IF another thread is ready at same priority as next_thread
18 IF next_thread holds a fast mutex
19 next_thread->iHeldFastMutex->iWaiting=TRUE
20 goto resched_end
22 next_thread->iTime=next_thread->iTimeslice (new timeslice)
23 next_thread=thread after next_thread in round-robin order
26 IF next_thread holds a fast mutex
27 IF next_thread holds system lock
28 goto resched_end
29 ELSE IF next_thread requires implicit system lock and system lock held
30 next_thread->iHeldFastMutex->iWaiting=TRUE
31 goto resched_end
34 IF next_thread is blocked on a fast mutex
35 IF next_thread->iWaitFastMutex->iHoldingThread (mutex is still locked)
36 next_thread=next_thread->iWaitFastMutex->iHoldingThread
37 goto resched_end
40 IF next_thread does not require implicit system lock
41 goto resched_end
42 ELSE IF system lock held by another thread
43 next_thread=system lock holding thread
44 system lock iWaiting = TRUE
46 resched_end:
47 iCurrentThread=next_thread
48 current stack pointer = next_thread->iSavedSP
49 switch_threads:
50 Restore CPU specific registers from stack
51 IF next_thread requires address space switch
52 Invoke address space switch handler (kernel still locked)
54 Restore nonvolatile registers from stack
55 Disable interrupts
56 IF IDFCs pending or another reschedule needed
57 goto start_resched
59 iKernCSLocked=0 (unlock the kernel)
60 Return with interrupts disabled

The main difference here is that we can invoke the address space switch handler with preemption disabled. This is because we can change address spaces in just a few assembly language instructions.

Scheduling of Symbian OS thread

As I said in the previous section, scheduling is the responsibility of the nanokernel, and the scheduler deals in nanothreads, not Symbian OS threads. Symbian OS only contributes to scheduling in the setting of thread priorities.

The Symbian OS thread class, DThread, has a member iThread-Priority. This specifies the priority in either absolute or process-relative form. Values between 0 and 63 inclusive represent absolute priorities (corresponding directly to nanokernel priorities) and negative values between -8 and -2 represent process-relative values (-1 is not used).

A call to the user-side API RThread::SetPriority() sets the iThreadPriority field using a value derived from the TThread-Priority argument that is passed to it. The kernel combines the iThreadPriority field with the process priority of the thread's owning process using a mapping table to produce the iDefaultPriority field. The following code shows how this is done.

Calculating Thread Priority

// Mapping table for thread+process priority to
// thread absolute priority
LOCAL_D const TUint8 ThreadPriorityTable[64] =
//Idle MuchLess Less Normal More MuchMore RealTime
/*Low*/ 1, 1, 2, 3, 4, 5, 22, 0,
/*Background*/ 3, 5, 6, 7, 8, 9, 22, 0,
/*Foreground*/ 3, 10, 11, 12, 13, 14, 22, 0,
/*High*/ 3, 17, 18, 19, 20, 22, 23, 0,
/*SystemServer1*/ 9, 15, 16, 21, 24, 25, 28, 0,
/*SystemServer2*/ 9, 15, 16, 21, 24, 25, 28, 0,
/*SystemServer3*/ 9, 15, 16, 21, 24, 25, 28, 0,
/*RealTimeServer*/ 18, 26, 27, 28, 29, 30, 31, 0
TInt DThread::CalcDefaultThreadPriority()
TInt r;
TInt tp=iThreadPriority;
if (tp>=0) // absolute thread priorities
if (tp<0)
TInt pp=iOwningProcess->iPriority; // proc priority 0-7
TInt i=(pp<<3)+tp;
// map thread+process priority to actual priority
return r;

This iDefaultPriority, returned from CalcDefaultThreadPriority() is the actual scheduling priority used by the thread when it doesn't hold a Symbian OS mutex, and so it is not subject to priority inheritance.

What about when the thread does hold a mutex? As we saw in Section, the nanokernel priority of a Symbian OS thread is given by the maximum of its iDefaultPriority value and the maximum priority of any entry on the thread's cleanup queue. Some of the cleanup queue entries result from mutexes held by the thread and, as we saw, the kernel adjusts their priorities to provide priority inheritance.

Scheduling in the emulator

As we saw in Chapter 1, Introducing EKA2, the emulator uses the host OS (Win32) threads. This means that the nanokernel needs to provide its own scheduling mechanism as Windows does not provide the 64 priority levels we have in Symbian OS. (Win32 only really provides five usable distinct priorities within a process.)

To make this work, we cannot allow Windows to arbitrarily schedule Symbian OS threads to run. The nanokernel achieves this by only making one Symbian OS thread ready to run as a Win32 thread; all the others will either be waiting on a Win32 event object or suspended. A side effect of this policy is that all Symbian OS threads can have the same standard Win32 priority.

Disabling preemption

EKA2 provides two mechanisms to disable preemption and rescheduling:

  1. Masking interrupts - disables interrupt dispatch and thus preemption
  2. Locking the kernel - disables preemption and rescheduling.

The emulator only has to emulate the first of these, since the second uses the same implementation as the target mobile phone.

We provide the interrupt mask in the emulator using something similar to a Win32 critical section object. Unlike the RCriticalSection of Symbian OS, this is re-entrant, allowing a single thread to wait on a critical section multiple times. The key difference to the standard Win32 critical section is the ability to operate correctly on a multi-processor PC in conjunction with the emulator's interrupt mechanism.

We disable interrupts by entering this critical section. This is effective as interrupts must do the same, and so they are blocked until the thread that owns the critical section releases it (by restoring interrupts). Thus this also prevents a thread being preempted whilst it has masked interrupts, as desired.


The kernel lock is just a simple counter, rather than the Win32 critical section used in EKA1; this allows the emulator to hand over the lock between threads when rescheduling exactly as the target scheduler does. The result is that the scheduler code follows the same algorithm as the nanokernel with regards to:

  • Disabling interrupts
  • Running IDFCs
  • Selecting a new thread
  • Exiting the scheduler.

The major difference in scheduling is in the handover code between threads. Any Symbian OS thread which is not the current thread will either be waiting on a Win32 event object, the reschedule lock, or suspended. Suspension is rare, and only occurs if the thread was preempted rather than voluntarily rescheduling. I discuss pre-emption in more detail in the next section; this section will concentrate on the normal case.

The handover from the old (executing) thread is usually done by signaling the new thread's reschedule lock and then waiting on the old one's lock. The new thread starts running from the same point in the code (because there is only one place where the waiting is done), the kernel is still locked and the algorithm continues as normal.

This is slightly different to the behavior on a real target phone. On the latter, there is no waiting, as the thread state is saved to its stack before the new thread state is restored. The scheduler knows that there is only one true thread of execution in the CPU and just changes the register context to effect the switch. But in the emulator we have two real execution contexts (ours, and the new thread) and we need to give the impression that execution is being directly handed off from one to the other.

Note that blocked threads nearly always have the same functions at the top of the call stack, SwitchThreads() and TScheduler::Reschedule().

Also note that this handover in the emulator works equally well when the new thread is the same as the old thread.

Interrupts and preemption

Interaction with the hardware in the emulator is always via the host OS. As a result, there are no real interrupts to handle in the emulator. Interaction with the host is always done using native threads, which make use of the host OS APIs for timer services, file I/O, UI and so on.

The emulator creates a number of Win32 threads to act as event or interrupt sources. These are not Symbian OS threads, therefore they are effectively unknown to the nanokernel scheduler. They run at a host priority above that of the Symbian OS threads to ensure that they respond to events immediately and cause preemption within the emulator.

The emulator provides two routines which act as the interrupt preamble and post-amble, StartOfInterrupt()and EndOfInterrupt(). The preamble routine, StartOfInterrupt(), disables interrupts and then suspends the host thread that is the current nanokernel thread. This ensures that the interrupt behaves like a real interrupt, executing while the current thread does nothing, even on multi-processor PCs. The post-amble routine, EndOfInterrupt(), does something similar to the ARM interrupt post-amble on a real phone - checking the kernel lock state and the need for a reschedule. If a reschedule is needed, the routine leaves the current thread suspended and causes a reschedule, otherwise it resumes the thread and enables interrupts again.

Unlike on a real phone, the end of interrupt function cannot cause the current thread to branch off to the scheduler because it is not safe to divert a Win32 thread from an arbitrary location - this can cause the entire host OS to hang. Instead we mark the current thread as preempted, leave it suspended, and signal another dedicated interrupt-rescheduler thread to do the reschedule on behalf of the preempted thread. This interrupt-rescheduler thread is a nanokernel thread that spends its entire life inside TScheduler::Reschedule() - it is not on the ready list and so the nanokernel never selects it to be run, but EndOfInterrupt() can wake it up when necessary to schedule the next thread.

When the nanokernel selects a preempted thread to run rather than signaling the reschedule lock, the emulator must resume the host thread. However, life is not that simple - there's more to do. On a phone, when the nanokernel schedules a thread, the kernel is locked. The kernel then makes a final check of the iDfcPending and iRescheduleNeededflags before unlocking and resuming where it left off. In the emulator, if we merely resume the preempted thread, then we will just do the final item in that list, and miss out the checks. This means that the current thread must do the checks (restarting the reschedule if required) and unlock the kernel on behalf of the preempted thread. This makes the handover to a preempted thread rather more complex than it is on the mobile phone.

The beauty of this method is that no side effects are apparent to users of the emulator. The Win32 version of the nanokernel has some pretty tricky code in it, but this ensures that the rest of the kernel just works - because it presents the same model as on real hardware. Idling

In the emulator, there is no way to idle the CPU from the Symbian OS null (idle) thread and the obvious alternative - going round in an infinite loop - is not very nice to the PC!

The null thread calls NThread::Idle() to go into idle mode on both the emulator and a phone. On the emulator, this sets a flag to indicate that the emulator is idle, and then waits on the thread's reschedule lock. The EndOfInterrupt() function detects this state as a special case and instead of using the interrupt-rescheduler thread, it just wakes up the null thread. The null thread then reschedules. Next time the null thread is scheduled, it returns from NThread::Idle() and so behaves in a similar fashion to its counterpart on real phones.

Dynamically loaded libraries

I have talked about threads and processes; I will finish this chapter with a short discussion on dynamically loaded libraries, or DLLs. You can find more on this subject in Chapter 8, Platform Security and Chapter 10, The Loader

The DLibrary class

The kernel creates a kernel-side library object (DLibrary) for every DLL that is explicitly loaded into a user process; that is, one that is the target of an RLibrary::Load() rather than one that is implicitly linked to by another executable. Library objects are specific to, and owned by, the process for which they were created; if two processes both load the same DLL, the kernel creates two separate DLibrary objects. A library has two main uses:

  1. It represents a link from a process to the global code graph. Each process always has at least one such connection - the DProcess::iCodeSeg pointer. This pointer, set up by the kernel at process load time, links each process to its own EXE code segment. DLibrary objects represent additional links to the code graph, created at run time.
  2. It provides a state machine to ensure that constructors and destructors for objects resident in .data and .bss sections are called correctly.

Libraries have two reference counts. One is the standard DObject reference count (since DLibrary derives from DObject); a non-zero value for this reference count simply stops the DLibrary itself being deleted - it does not stop the underlying code segment being deleted or removed from any process.

The second reference count (iMapCount) is the number of user references on the library, which is equal to the number of handles on the library opened by the process or by any of its threads. The kernel always updates this count with the CodeSegLockmutex held. When the last user handle is closed, iMapCount will reach zero and this triggers the calling of static destructors and the removal of the library code segment from the process address space.

The loader creates DLibrary objects on behalf of a client loading a DLL. A process may not have more than one DLibrary referring to the same code segment. If a process loads the same library twice, the kernel will open a second handle for it on the already existing DLibrary and its map count will be incremented.

A DLibrary object transitions through the following states during its life:

  • ECreated - transient state in which object is created. Switches to ELoaded or EAttached when library and corresponding code segment are added to the target process
  • ELoaded - code segment is loaded and attached to the target process but the kernel has not called static constructors
  • EAttaching - the target process is currently running the code segment static constructors. Transitions to EAttached when constructors have completed
  • EAttached - static constructors have completed and the code segment is fully available for use by the target process
  • EDetachPending - the last user handle has been closed on the DLibrary but static destructors have not yet been called. Transitions to EDetaching just before running static destructors
  • EDetaching - the target process is currently running the code segment static destructors. Transitions to ELoaded when destructors have completed.

Let's have a look at the Dlibrary class:

class DLibrary : public DObject 
enum TState
ECreated=0, // initial state
ELoaded=1, // code segment loaded
EAttaching=2, // calling constructors
EAttached=3, // all constructors done
EDetachPending=4, // about to call destructors
EDetaching=5, // calling destructors
static TInt New(DLibrary*& aLib, DProcess* aProcess, DCodeSeg* aSeg);
void RemoveFromProcess();
virtual ~DLibrary();
virtual TInt Close(TAny* aPtr);
virtual TInt AddToProcess(DProcess* aProcess);
virtual void DoAppendName(TDes& aName);
TInt iMapCount;
TUint8 iState;
SDblQueLink iThreadLink; // attaches to opening/closing thread
DCodeSeg* iCodeSeg;

Key member data of DLibrary

Field Description
iMapCount Count of the number of times this library is mapped into the process; equal to the number of user handles the process and its threads have on this library.
iState Records progress in calling user-side constructors or destructors during library loading and unloading.
iThreadLink Doubly linked list field, which is used to attach the library to the thread that is running user-side constructors or destructors. This is needed to enable cleanup if the thread terminates while running one of these functions.
iCodeSeg Pointer to the code segment to which this library refers.

Code segments

I mentioned that a library represents a link from a process to the global code graph, and that this link is held in the iCodeSeg pointer. This pointer denotes a kernel object known as a DCodeSeg.

A DCodeSeg is an object that represents the contents of an executable, relocated for particular code and data addresses. Executable programs (EXEs) or dynamically loaded libraries (DLLs), execute in place (XIP) or RAM-loaded - whatever the combination, the code is represented by a DCodeSeg.

On EKA1, EXEs and DLLs were handled separately and differently, which gave us problems when we came to load a DLL that linked back to an EXE.

Under EKA2, the unification of the support for loading code under the DCodeSeg has made matters much simpler. Multiple instances of the same process will use the same DCodeSeg unless a RAM-loaded fixed process needs different data addresses. Similarly, if a DLL is loaded into several processes, the same DCodeSeg is attached to all the processes (the code is shared), unless different data addresses are required. This happens when a RAM-loaded DLL with writable static data is then loaded into more than one fixed process, or into a combination of fixed and non-fixed processes.

I will discuss this in greater depth in Chapter 10, The Loader.


In this chapter I have talked about threads in the nanokernel, and Symbian OS threads, processes and libraries. Of course, these are fundamental operating system concepts, and you will find that they form an important basis for other parts of this book.

Licence icon cc-by-sa 3.0-88x31.png© 2010 Symbian Foundation Limited. This document is licensed under the Creative Commons Attribution-Share Alike 2.0 license. See for the full terms of the license.
Note that this content was originally hosted on the Symbian Foundation developer wiki.

This page was last modified on 6 May 2013, at 10:58.
73 page views in the last 30 days.

Was this page helpful?

Your feedback about this content is important. Let us know what you think.


Thank you!

We appreciate your feedback.