Some Flags About Workqueue in Kernel

What are these flags for workqueue means?

WQ_DRAINING

This flag is used to indicate that the kernel is currently flushing the workqueue and new work items
cannot be queued on it. Only currently pending or running work items are allowed to so during this phase until the entire workqueue is completely empty.

For details, check out the implementation of drain_workqueue() in kernel/workqueue.c.

---

WQ_RESCUER

This flag is already deprecated in the latest Kernel by this patch and the behaviour is now determined by the WQ_MEM_RECLAIM flag.

As far as the "rescuer" functionality is concerned, here is the relevant section of documentation from kernel/workqueue.c,

Workqueue rescuer thread function. There's one rescuer for each
workqueue which has WQ_MEM_RECLAIM set.

Regular work processing on a pool may block trying to create a new
worker which uses GFP_KERNEL allocation which has slight chance of
developing into deadlock if some works currently on the same queue
need to be processed to satisfy the GFP_KERNEL allocation. This is
the problem rescuer solves.

When such condition is possible, the pool summons rescuers of all
workqueues which have works queued on the pool and let them process
those works so that forward progress can be guaranteed.

---

WQ_MAX_UNBOUND_PER_CPU

_{(Contrary to how you have interpreted it, WQ_MAX_UNBOUND_PER_CPU is NOT the number of cpus. It is the number of workqueues that can be associated with a cpu.)}

Workqueues have been traditionally per-cpu i.e. each workqueue was associated with a particular cpu, resulting in better performance due to cache-locality. The kernel scheduler does NOT have a choice but to schedule it always on the cpu it was defined on. On current architectures, this leads to increased power consumption as even a single workqueue can prevent a cpu from idling and being turned off. Hence unbound workqueues have been introduced. The scheduler is free to re-schedule unbound workqueues on any cpu as it sees fit.

The total number of such workqueues is limited to WQ_UNBOUND_MAX_ACTIVE which is defined as num_possible_cpus() * WQ_MAX_UNBOUND_PER_CPU (upto a limit of total workqueues in the system determined by WQ_MAX_ACTIVE).

Kernel Oops when calling aio_complete by workqueue

In your async_read_work function, you call kfree(a_work);, set a_work = NULL; and then dereference a_work here: a_work->a_iocb->ki_complete(a_work->a_iocb,100, 0);.

Possible solutions:

1. Rearrange the code:

       a_work->a_iocb->ki_complete(a_work->a_iocb,100, 0);
       kfree(a_work);
       a_work = NULL; /* not really necessary */
   

2. Use a local variable:

       struct kiocb *iocb;
       ...
       iocb = a_work->a_iocb;
       kfree(a_work);
       a_work = NULL; /* not really necessary */
       iocb->ki_complete(iocb,100, 0);
   

wait queues and work queues, do they always go together?

As part of the probe callback, this driver initializes a work queue which is part of the driver’s private structure and adds itself to the Q. But I do not see any blocking of any kind anywhere.

I think you meant the wait queue head, not the work queue. I do not see any evidence of the probe adding itself to the queue; it is merely initializing the queue.

The queue is used by the calls to the wait_event_timeout() macro in the bcmgenet_mii_read() and bcmgenet_mii_write() functions in bcmmii.c. These calls will block until either the condition they are waiting for becomes true or the timeout period elapses. They are woken up by the wake_up(&priv->wq); call in the ISR0 interrupt handler.

Then it goes on to initialize a work queue with a function to call when woken up.

It is initializing a work item, not a work queue. The function will be called from a kernel thread as a result of the work item being added to the system work queue.

Now coming to the ISR0 for the driver, within that is an explicit call to the scheduler as part of the ISR (bcmgenet_isr0) if certain conditions are met. Now AFAIK, this call is used to defer work to a later time, much like a tasklet does.

You are referring to the schedule_work(&priv->bcmgenet_irq_work); call in the ISR0 interrupt handler. This is adding the previously mentioned work item to the system work queue. It is similar to as tasklet, but tasklets are run in a softirq context whereas work items are run in a process context.

Post this we check some MDIO status flags and if the conditions are met, we wake up the process which was blocked in process context. But where exactly is the process blocked?

As mentioned above, the process is blocked in the bcmgenet_mii_read() and bcmgenet_mii_write() functions, although they use a timeout to avoid blocking for long periods. (This timeout is especially important for those versions of GENET that do not support MDIO-related interrupts!)

Also, most of the time, wait queues seem to be used in conjunction with work queues. Is that the typical way to use them?

Not especially. This particular driver uses both a wait queue and a work item, but I wouldn't describe them as being used "in conjunction" since they are being used to handle different interrupt conditions.

How to queue the same workqueue work multiple times in Linux?

Workqueue expects every work structure to represent single "task", which is needed to be run once.

So, then simplest way to run a task several times - create new work structure every time.

Alternatively, as repeating the work while it is running is something unusual for workqueue, you may create your own kernel thread for execute some function repeatedly:

DECLARE_WAITQUEUE(repeat_wq); // Kernel thread will wait on this workqueue.
int n_works = 0; // Number of work requests to process.

// Thread function
void repeat_work(void* unused)
{
    spin_lock_irq(repeat_wq.lock); // Reuse workqueue's spinlock for our needs
    while(1) {
        // Wait until work request or thread should be stopped
        wait_event_interruptible_locked(&repeat_wq,
            n_works || kthread_should_stop());
        if(kthread_should_stop()) break;
        spin_unlock_irq(repeat_wq.lock);

        <do the work>

        // Acquire the lock for decrement count and recheck condition
        spin_lock_irq(repeat_wq.lock);
        n_works--;
    }
    // Finally release the lock
    spin_unlock_irq(repeat_wq.lock);
}

// Request new work.
void add_work(void)
{
    unsigned long flags;
    spin_lock_irqsave(repeat_wq.lock, flags);
    n_works++;
    wake_up_locked(&repeat_wq);
    spin_unlock_irqrestore(repeat_wq.lock, flags);      
}

Workqueues are kernel threads too, with a specific thread function kthread_worker_fn().

---

Related Topics