When I readup on spinlocks, it seems like I need to choose between disabling interrupts and not. If a spinlock_t is never used during an interrupt, am I safe to leave interrupts enabled while I hold the lock? (Same question for read/write locks if it is different.) -Kai Meyer
Hi Kai, On Wed, Nov 9, 2011 at 1:07 PM, Kai Meyer <kai@gnukai.com> wrote:
When I readup on spinlocks, it seems like I need to choose between disabling interrupts and not. If a spinlock_t is never used during an interrupt, am I safe to leave interrupts enabled while I hold the lock? (Same question for read/write locks if it is different.)
So the intention behind using a spinlock is to provide mutual exclusion. A spinlock by itself only really provides mutual exclusion between 2 cores, and not within the same core. To provide the mutual exclusion within the same core, you need to disable interrupts. Normally, you would disable interrupts and acquire the spinlock to guarantee that mutual exclusion, and the only reason you would normally use the spinlock without disabling interrupts is when you know that interrupts are already disabled. The danger of acquiring a spinlock with interrupts enabled is that if another interrupt fired (or the same interrupt fired again) and it tried to acquire the same spinlock, then you could have deadlock. If no interrupts touch the spinlock, then you're probably using the wrong mutual exclusion mechanism. spinlocks are really intended to provide mutual exclsion between interrupt context and non-interrupt context. Also remember, that on a non-SMP (aka UP) build, spinlocks become no-ops (except when certain debug checking code is enabled). -- Dave Hylands Shuswap, BC, Canada http://www.davehylands.com
Ok, I need mutual exclusion on a data structure regardless of interrupts and core. It sounds like it can be done by using a spinlock and disabling interrupts, but you mention that "spinlocks are intended to provide mutual exclsion between interrupt context and non-interrupt context." Should I be using a semaphore (mutex) instead? Perhaps I could explain my problem with some code: struct my_struct *get_data(spinlock_t *mylock, int ALLOC_DATA) { struct my_struct *mydata = NULL; spin_lock(mylock); if (test_bit(index, mybitmap)) mydata = retrieve_data(); if (!mydata && ALLOC_DATA) { mydata = alloc_data(); set_bit(index, mybitmap); } spin_unlock(mylock); return mydata; } I need to prevent retrieve_data from being called if the index bit is set in mybitmap and alloc_data has not completed, so I use a bitmap to indicate that alloc_data has completed. I also need to protect alloc_data from being run multiple times, so I use the spin_lock to ensure that test_bit (and possibly retrieve_data) is not run while alloc_data is being run (because it runs while the bit is cleared). -Kai Meyer On 11/09/2011 02:40 PM, Dave Hylands wrote:
Hi Kai,
On Wed, Nov 9, 2011 at 1:07 PM, Kai Meyer<kai@gnukai.com> wrote:
When I readup on spinlocks, it seems like I need to choose between disabling interrupts and not. If a spinlock_t is never used during an interrupt, am I safe to leave interrupts enabled while I hold the lock? (Same question for read/write locks if it is different.) So the intention behind using a spinlock is to provide mutual exclusion.
A spinlock by itself only really provides mutual exclusion between 2 cores, and not within the same core. To provide the mutual exclusion within the same core, you need to disable interrupts.
Normally, you would disable interrupts and acquire the spinlock to guarantee that mutual exclusion, and the only reason you would normally use the spinlock without disabling interrupts is when you know that interrupts are already disabled.
The danger of acquiring a spinlock with interrupts enabled is that if another interrupt fired (or the same interrupt fired again) and it tried to acquire the same spinlock, then you could have deadlock.
If no interrupts touch the spinlock, then you're probably using the wrong mutual exclusion mechanism. spinlocks are really intended to provide mutual exclsion between interrupt context and non-interrupt context.
Also remember, that on a non-SMP (aka UP) build, spinlocks become no-ops (except when certain debug checking code is enabled).
-----Original Message----- From: kernelnewbies-bounces+jharan=bytemobile.com@kernelnewbies.org [mailto:kernelnewbies- bounces+jharan=bytemobile.com@kernelnewbies.org] On Behalf Of Kai Meyer Sent: Wednesday, November 09, 2011 3:12 PM To: kernelnewbies@kernelnewbies.org Subject: Re: Spinlocks and interrupts
Ok, I need mutual exclusion on a data structure regardless of interrupts and core. It sounds like it can be done by using a spinlock and disabling interrupts, but you mention that "spinlocks are intended to provide mutual exclsion between interrupt context and non-interrupt context." Should I be using a semaphore (mutex) instead?
Perhaps I could explain my problem with some code: struct my_struct *get_data(spinlock_t *mylock, int ALLOC_DATA) { struct my_struct *mydata = NULL; spin_lock(mylock); if (test_bit(index, mybitmap)) mydata = retrieve_data(); if (!mydata && ALLOC_DATA) { mydata = alloc_data(); set_bit(index, mybitmap); } spin_unlock(mylock); return mydata; }
I need to prevent retrieve_data from being called if the index bit is set in mybitmap and alloc_data has not completed, so I use a bitmap to indicate that alloc_data has completed. I also need to protect alloc_data from being run multiple times, so I use the spin_lock to ensure that test_bit (and possibly retrieve_data) is not run while alloc_data is being run (because it runs while the bit is cleared).
-Kai Meyer
You probably want to lock with spin_lock_irqsave() and unlock with spin_unlock_irqrestore() if you are not sure about what contexts get_data() will be called in. There's plenty of examples of how to use these in the kernel sources. I note you are passing in the address of the spinlock itself. Be wary of deadly embraces where two threads acquire two locks in different order. Jeff Haran
Hi Kai, On Wed, Nov 9, 2011 at 3:12 PM, Kai Meyer <kai@gnukai.com> wrote:
Ok, I need mutual exclusion on a data structure regardless of interrupts and core. It sounds like it can be done by using a spinlock and disabling interrupts, but you mention that "spinlocks are intended to provide mutual exclsion between interrupt context and non-interrupt context." Should I be using a semaphore (mutex) instead?
It depends. If the function is only called from thread context, then you probably want to use a mutex. If there is a possibility that it might be called from interrupt context, then you can't use a mutex. Also, remember that spin-locks are no-ops on a single processor machine, so as coded, you have no protection on a single-processor machine if you're calling from thread context.
Perhaps I could explain my problem with some code: struct my_struct *get_data(spinlock_t *mylock, int ALLOC_DATA) { struct my_struct *mydata = NULL; spin_lock(mylock); if (test_bit(index, mybitmap)) mydata = retrieve_data(); if (!mydata && ALLOC_DATA) { mydata = alloc_data(); set_bit(index, mybitmap); } spin_unlock(mylock); return mydata; }
I need to prevent retrieve_data from being called if the index bit is set in mybitmap and alloc_data has not completed, so I use a bitmap to indicate that alloc_data has completed. I also need to protect alloc_data from being run multiple times, so I use the spin_lock to ensure that test_bit (and possibly retrieve_data) is not run while alloc_data is being run (because it runs while the bit is cleared).
If alloc_data might block, then you can't disable interrupts and you definitely shouldn't be using spinlocks. -- Dave Hylands Shuswap, BC, Canada http://www.davehylands.com
On 11/09/2011 08:38 PM, Dave Hylands wrote:
Hi Kai,
On Wed, Nov 9, 2011 at 3:12 PM, Kai Meyer<kai@gnukai.com> wrote:
Ok, I need mutual exclusion on a data structure regardless of interrupts and core. It sounds like it can be done by using a spinlock and disabling interrupts, but you mention that "spinlocks are intended to provide mutual exclsion between interrupt context and non-interrupt context." Should I be using a semaphore (mutex) instead? It depends. If the function is only called from thread context, then you probably want to use a mutex. If there is a possibility that it might be called from interrupt context, then you can't use a mutex.
Also, remember that spin-locks are no-ops on a single processor machine, so as coded, you have no protection on a single-processor machine if you're calling from thread context.
To make sure I understand you, it sounds like there's two contexts I need to be concerned about, thread context and interrupt context. As far as I can be sure, this code will only run in thread context. If you could verify for me that a block device's make request function is only reached in thread context, then that would make me doubly sure.
Perhaps I could explain my problem with some code: struct my_struct *get_data(spinlock_t *mylock, int ALLOC_DATA) { struct my_struct *mydata = NULL; spin_lock(mylock); if (test_bit(index, mybitmap)) mydata = retrieve_data(); if (!mydata&& ALLOC_DATA) { mydata = alloc_data(); set_bit(index, mybitmap); } spin_unlock(mylock); return mydata; }
I need to prevent retrieve_data from being called if the index bit is set in mybitmap and alloc_data has not completed, so I use a bitmap to indicate that alloc_data has completed. I also need to protect alloc_data from being run multiple times, so I use the spin_lock to ensure that test_bit (and possibly retrieve_data) is not run while alloc_data is being run (because it runs while the bit is cleared). If alloc_data might block, then you can't disable interrupts and you definitely shouldn't be using spinlocks.
alloc_data will call kmalloc(size, GFP_KERNEL), which I think may block, so disabling irqs is out. Between thread context and kmalloc with GFP_KERNEL, it sounds like your suggestion would be to use a mutex. Is that correct? -Kai Meyer
Well, I changed my code to use a mutex instead of a spinlock, and now I get: BUG: scheduling while atomic: swapper/0/0x10010000 All I changed was the spinlock_t to a struct mutex, and call mutex_init, mutex_lock, and mutex_unlock where I was previously calling the spin_lock variations. I'm confused. What does mutex_lock do besides set values in an atomic_t? -Kai Meyer On 11/10/2011 10:02 AM, Kai Meyer wrote:
On 11/09/2011 08:38 PM, Dave Hylands wrote:
Hi Kai,
On Wed, Nov 9, 2011 at 3:12 PM, Kai Meyer<kai@gnukai.com> wrote:
Ok, I need mutual exclusion on a data structure regardless of interrupts and core. It sounds like it can be done by using a spinlock and disabling interrupts, but you mention that "spinlocks are intended to provide mutual exclsion between interrupt context and non-interrupt context." Should I be using a semaphore (mutex) instead? It depends. If the function is only called from thread context, then you probably want to use a mutex. If there is a possibility that it might be called from interrupt context, then you can't use a mutex.
Also, remember that spin-locks are no-ops on a single processor machine, so as coded, you have no protection on a single-processor machine if you're calling from thread context.
To make sure I understand you, it sounds like there's two contexts I need to be concerned about, thread context and interrupt context. As far as I can be sure, this code will only run in thread context. If you could verify for me that a block device's make request function is only reached in thread context, then that would make me doubly sure.
Perhaps I could explain my problem with some code: struct my_struct *get_data(spinlock_t *mylock, int ALLOC_DATA) { struct my_struct *mydata = NULL; spin_lock(mylock); if (test_bit(index, mybitmap)) mydata = retrieve_data(); if (!mydata&& ALLOC_DATA) { mydata = alloc_data(); set_bit(index, mybitmap); } spin_unlock(mylock); return mydata; }
I need to prevent retrieve_data from being called if the index bit is set in mybitmap and alloc_data has not completed, so I use a bitmap to indicate that alloc_data has completed. I also need to protect alloc_data from being run multiple times, so I use the spin_lock to ensure that test_bit (and possibly retrieve_data) is not run while alloc_data is being run (because it runs while the bit is cleared). If alloc_data might block, then you can't disable interrupts and you definitely shouldn't be using spinlocks.
alloc_data will call kmalloc(size, GFP_KERNEL), which I think may block, so disabling irqs is out.
Between thread context and kmalloc with GFP_KERNEL, it sounds like your suggestion would be to use a mutex. Is that correct?
-Kai Meyer
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
I think I get it. I'm hitting the scheduling while atomic because I'm calling my function from a struct bio's endio function, which is probably running with a lock held somewhere else, and then my mutex sleeps, while the spin_lock functions do not sleep. Perhaps I need to learn more about the context in which my endio function is being called. On 11/10/2011 11:02 AM, Kai Meyer wrote:
Well, I changed my code to use a mutex instead of a spinlock, and now I get: BUG: scheduling while atomic: swapper/0/0x10010000 All I changed was the spinlock_t to a struct mutex, and call mutex_init, mutex_lock, and mutex_unlock where I was previously calling the spin_lock variations. I'm confused. What does mutex_lock do besides set values in an atomic_t?
-Kai Meyer
On 11/10/2011 10:02 AM, Kai Meyer wrote:
On 11/09/2011 08:38 PM, Dave Hylands wrote:
Hi Kai,
On Wed, Nov 9, 2011 at 3:12 PM, Kai Meyer<kai@gnukai.com> wrote:
Ok, I need mutual exclusion on a data structure regardless of interrupts and core. It sounds like it can be done by using a spinlock and disabling interrupts, but you mention that "spinlocks are intended to provide mutual exclsion between interrupt context and non-interrupt context." Should I be using a semaphore (mutex) instead? It depends. If the function is only called from thread context, then you probably want to use a mutex. If there is a possibility that it might be called from interrupt context, then you can't use a mutex.
Also, remember that spin-locks are no-ops on a single processor machine, so as coded, you have no protection on a single-processor machine if you're calling from thread context.
To make sure I understand you, it sounds like there's two contexts I need to be concerned about, thread context and interrupt context. As far as I can be sure, this code will only run in thread context. If you could verify for me that a block device's make request function is only reached in thread context, then that would make me doubly sure.
Perhaps I could explain my problem with some code: struct my_struct *get_data(spinlock_t *mylock, int ALLOC_DATA) { struct my_struct *mydata = NULL; spin_lock(mylock); if (test_bit(index, mybitmap)) mydata = retrieve_data(); if (!mydata&& ALLOC_DATA) { mydata = alloc_data(); set_bit(index, mybitmap); } spin_unlock(mylock); return mydata; }
I need to prevent retrieve_data from being called if the index bit is set in mybitmap and alloc_data has not completed, so I use a bitmap to indicate that alloc_data has completed. I also need to protect alloc_data from being run multiple times, so I use the spin_lock to ensure that test_bit (and possibly retrieve_data) is not run while alloc_data is being run (because it runs while the bit is cleared). If alloc_data might block, then you can't disable interrupts and you definitely shouldn't be using spinlocks.
alloc_data will call kmalloc(size, GFP_KERNEL), which I think may block, so disabling irqs is out.
Between thread context and kmalloc with GFP_KERNEL, it sounds like your suggestion would be to use a mutex. Is that correct?
-Kai Meyer
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Hi Kai, On Thu, Nov 10, 2011 at 10:14 AM, Kai Meyer <kai@gnukai.com> wrote:
I think I get it. I'm hitting the scheduling while atomic because I'm calling my function from a struct bio's endio function, which is probably running with a lock held somewhere else, and then my mutex sleeps, while the spin_lock functions do not sleep.
Actually, just holding a lock doesn't create an atomic context. Disabling interrupts, and maybe disabling preemption will create atomic contexts. An atomic context is basically a context where you're not allowed to call blocking functions. Use of spinlocks is pretty much your only option in this scenario (unless of course you can somehow do what you want outside of the atomic context).
Perhaps I need to learn more about the context in which my endio function is being called.
I think so. I haven't written any block drivers, so I'm not familiar with all the gory details. What the errors you're getting tell me is that you function is sometimes called from an atomic context. This means that you shouldn't be calling kmalloc, unless you use the GFP_ATOMIC option, and you need to be prepared to deal with getting a NULL pointer back (which doesn't necessarily mean that you've run out of memory, just that you would have to block in order to allocate the memory). Generally speaking, at least with character drivers, I try to all of my allocations while I'm in thread context (like during an open or ioctl call). -- Dave Hylands Shuswap, BC, Canada http://www.davehylands.com
-----Original Message----- From: kernelnewbies-bounces+jharan=bytemobile.com@kernelnewbies.org [mailto:kernelnewbies- bounces+jharan=bytemobile.com@kernelnewbies.org] On Behalf Of Dave Hylands Sent: Thursday, November 10, 2011 11:07 AM To: Kai Meyer Cc: kernelnewbies@kernelnewbies.org Subject: Re: Spinlocks and interrupts
Hi Kai,
On Thu, Nov 10, 2011 at 10:14 AM, Kai Meyer <kai@gnukai.com> wrote:
I think I get it. I'm hitting the scheduling while atomic because I'm calling my function from a struct bio's endio function, which is probably running with a lock held somewhere else, and then my mutex sleeps, while the spin_lock functions do not sleep.
Actually, just holding a lock doesn't create an atomic context.
I believe on kernels with kernel pre-emption enabled the act of taking the lock disables pre-emption. If it didn't work this way you could end up taking the lock in one process context and while the lock was held get pre-empted. Then another process tries to take the lock and you dead lock. Jeff Haran
Alright, to summarize, for my benefit mostly, I'm writing a block device driver, which has 2 entry points into my code that will reach this critical section. It's either the make request function for the block device, or the resulting bio->bi_end_io function. I do some waiting with msleep() (for now) from the make request function entry point, so I'm confident that entry point is not in an atomic context. I also only end up requesting the critical section to call kmalloc from this context, which is why I never ran into the scheduling while atomic issue before. I'm fairly certain the critical section executes in thread context not interrupt context from either entry point. I'm certain that the spinlock_t is only ever used in one function (a I posted a simplified version of the critical section earlier). It seems that the critical section is often called in an atomic context. The spin_lock function sounds like it will only cause a second call to spin_lock to spin if it is called on a separate core. But, since I'm certain the critical section is never called from interrupt context, only thread context, the fact that pre-emption is disabled on the core should provide the protection I need with out having to disable IRQs. Disabling IRQs would prevent an interrupt from occurring while the lock is acquired. I would like to avoid disabling interrupts if I don't need to. So it sounds like spin_lock/spin_unlock is the correct choice? In addition, I'd like to be more confident in my assumptions above. Can I test for atomic context? For instance, I know that you can call irqs_disabled(), is there a similar is_atomic() function I can call? I would like to put a few calls in different places to learn what sort of context I'm. -Kai Meyer On 11/10/2011 12:19 PM, Jeff Haran wrote:
-----Original Message----- From: kernelnewbies-bounces+jharan=bytemobile.com@kernelnewbies.org [mailto:kernelnewbies- bounces+jharan=bytemobile.com@kernelnewbies.org] On Behalf Of Dave Hylands Sent: Thursday, November 10, 2011 11:07 AM To: Kai Meyer Cc: kernelnewbies@kernelnewbies.org Subject: Re: Spinlocks and interrupts
Hi Kai,
On Thu, Nov 10, 2011 at 10:14 AM, Kai Meyer<kai@gnukai.com> wrote:
I think I get it. I'm hitting the scheduling while atomic because I'm calling my function from a struct bio's endio function, which is probably running with a lock held somewhere else, and then my mutex sleeps, while the spin_lock functions do not sleep. Actually, just holding a lock doesn't create an atomic context. I believe on kernels with kernel pre-emption enabled the act of taking the lock disables pre-emption. If it didn't work this way you could end up taking the lock in one process context and while the lock was held get pre-empted. Then another process tries to take the lock and you dead lock.
Jeff Haran
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
-----Original Message----- From: kernelnewbies-bounces@kernelnewbies.org [mailto:kernelnewbies- bounces@kernelnewbies.org] On Behalf Of Kai Meyer Sent: Thursday, November 10, 2011 1:55 PM To: kernelnewbies@kernelnewbies.org Subject: Re: Spinlocks and interrupts
Alright, to summarize, for my benefit mostly,
I'm writing a block device driver, which has 2 entry points into my code that will reach this critical section. It's either the make request function for the block device, or the resulting bio->bi_end_io function. I do some waiting with msleep() (for now) from the make request function entry point, so I'm confident that entry point is not in an atomic context. I also only end up requesting the critical section to call kmalloc from this context, which is why I never ran into the scheduling while atomic issue before.
I'm fairly certain the critical section executes in thread context not interrupt context from either entry point.
I'm certain that the spinlock_t is only ever used in one function (a I posted a simplified version of the critical section earlier).
It seems that the critical section is often called in an atomic context.
The spin_lock function sounds like it will only cause a second call to spin_lock to spin if it is called on a separate core.
But, since I'm certain the critical section is never called from interrupt context, only thread context, the fact that pre-emption is disabled on the core should provide the protection I need with out having to disable IRQs. Disabling IRQs would prevent an interrupt from occurring while the lock is acquired. I would like to avoid disabling interrupts if I don't need to.
So it sounds like spin_lock/spin_unlock is the correct choice?
In addition, I'd like to be more confident in my assumptions above. Can I test for atomic context? For instance, I know that you can call irqs_disabled(), is there a similar is_atomic() function I can call? I would like to put a few calls in different places to learn what sort of context I'm.
-Kai Meyer
On 11/10/2011 12:19 PM, Jeff Haran wrote:
-----Original Message----- From: kernelnewbies- bounces+jharan=bytemobile.com@kernelnewbies.org [mailto:kernelnewbies- bounces+jharan=bytemobile.com@kernelnewbies.org] On Behalf Of Dave Hylands Sent: Thursday, November 10, 2011 11:07 AM To: Kai Meyer Cc: kernelnewbies@kernelnewbies.org Subject: Re: Spinlocks and interrupts
Hi Kai,
On Thu, Nov 10, 2011 at 10:14 AM, Kai Meyer<kai@gnukai.com> wrote:
I think I get it. I'm hitting the scheduling while atomic because I'm calling my function from a struct bio's endio function, which is probably running with a lock held somewhere else, and then my mutex sleeps, while the spin_lock functions do not sleep. Actually, just holding a lock doesn't create an atomic context. I believe on kernels with kernel pre-emption enabled the act of taking the lock disables pre-emption. If it didn't work this way you could end up taking the lock in one process context and while the lock was held get pre-empted. Then another process tries to take the lock and you dead lock.
Jeff Haran
Kai, you might want to try bottom posting. It is the standard on these lists. It makes it easier for others to follow the thread. I know of no kernel call that you can make to test for current execution context. There are the in_irq(), in_interrupt() and in_softirq() macros in hardirq.h, but when I've looked at the code that implements them I've come to the conclusion that they sometimes will lie. in_softirq() returns non-zero if you are in a software IRQ. Fair enough. But based on my reading in the past it's looked to me like it will also return non-zero if you've disabled bottom halves from process context with say a call to spin_lock_bh(). It would be nice if there were some way of asking the kernel what context you are in, for debugging if for no other reason, but if it's there I haven't found it. I'd love to be proven wrong here, BTW. If others know better, please enlighten me. Jeff Haran
On 11/10/2011 04:00 PM, Jeff Haran wrote:
-----Original Message----- From: kernelnewbies-bounces@kernelnewbies.org [mailto:kernelnewbies- bounces@kernelnewbies.org] On Behalf Of Kai Meyer Sent: Thursday, November 10, 2011 1:55 PM To: kernelnewbies@kernelnewbies.org Subject: Re: Spinlocks and interrupts
Alright, to summarize, for my benefit mostly,
I'm writing a block device driver, which has 2 entry points into my code that will reach this critical section. It's either the make request function for the block device, or the resulting bio->bi_end_io function. I do some waiting with msleep() (for now) from the make request function entry point, so I'm confident that entry point is not in an atomic context. I also only end up requesting the critical section to call kmalloc from this context, which is why I never ran into the scheduling while atomic issue before.
I'm fairly certain the critical section executes in thread context not interrupt context from either entry point.
I'm certain that the spinlock_t is only ever used in one function (a I posted a simplified version of the critical section earlier).
It seems that the critical section is often called in an atomic context. The spin_lock function sounds like it will only cause a second call to spin_lock to spin if it is called on a separate core.
But, since I'm certain the critical section is never called from interrupt context, only thread context, the fact that pre-emption is disabled on the core should provide the protection I need with out having to disable IRQs. Disabling IRQs would prevent an interrupt from occurring while the lock is acquired. I would like to avoid disabling interrupts if I don't need to.
So it sounds like spin_lock/spin_unlock is the correct choice?
In addition, I'd like to be more confident in my assumptions above. Can I test for atomic context? For instance, I know that you can call irqs_disabled(), is there a similar is_atomic() function I can call? I would like to put a few calls in different places to learn what sort of context I'm.
-Kai Meyer
On 11/10/2011 12:19 PM, Jeff Haran wrote:
-----Original Message----- From: kernelnewbies- bounces+jharan=bytemobile.com@kernelnewbies.org [mailto:kernelnewbies- bounces+jharan=bytemobile.com@kernelnewbies.org] On Behalf Of Dave Hylands Sent: Thursday, November 10, 2011 11:07 AM To: Kai Meyer Cc: kernelnewbies@kernelnewbies.org Subject: Re: Spinlocks and interrupts
Hi Kai,
On Thu, Nov 10, 2011 at 10:14 AM, Kai Meyer<kai@gnukai.com> wrote:
I think I get it. I'm hitting the scheduling while atomic because I'm calling my function from a struct bio's endio function, which is probably running with a lock held somewhere else, and then my mutex sleeps, while the spin_lock functions do not sleep. Actually, just holding a lock doesn't create an atomic context. I believe on kernels with kernel pre-emption enabled the act of taking the lock disables pre-emption. If it didn't work this way you could end up taking the lock in one process context and while the lock was held get pre-empted. Then another process tries to take the lock and you dead lock.
Jeff Haran
Kai, you might want to try bottom posting. It is the standard on these lists. It makes it easier for others to follow the thread.
I know of no kernel call that you can make to test for current execution context. There are the in_irq(), in_interrupt() and in_softirq() macros in hardirq.h, but when I've looked at the code that implements them I've come to the conclusion that they sometimes will lie. in_softirq() returns non-zero if you are in a software IRQ. Fair enough. But based on my reading in the past it's looked to me like it will also return non-zero if you've disabled bottom halves from process context with say a call to spin_lock_bh().
It would be nice if there were some way of asking the kernel what context you are in, for debugging if for no other reason, but if it's there I haven't found it.
I'd love to be proven wrong here, BTW. If others know better, please enlighten me.
Jeff Haran
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
I try to remember to bottom post on message lists, but obviously I've been negligent :) Perhaps I'll just add some calls to msleep() at various places to help me identify when portions of my code are in an atomic context, just to help me learn what's going on. -Kai Meyer
For most of the block drivers bio_endio runs in a context of its tasklet, so it is indeed atomic context. -Rajat On Fri, Nov 11, 2011 at 4:50 AM, Kai Meyer <kai@gnukai.com> wrote:
On 11/10/2011 04:00 PM, Jeff Haran wrote:
-----Original Message----- From: kernelnewbies-bounces@kernelnewbies.org [mailto:kernelnewbies- bounces@kernelnewbies.org] On Behalf Of Kai Meyer Sent: Thursday, November 10, 2011 1:55 PM To: kernelnewbies@kernelnewbies.org Subject: Re: Spinlocks and interrupts
Alright, to summarize, for my benefit mostly,
I'm writing a block device driver, which has 2 entry points into my code that will reach this critical section. It's either the make request function for the block device, or the resulting bio->bi_end_io function. I do some waiting with msleep() (for now) from the make request function entry point, so I'm confident that entry point is not in an atomic context. I also only end up requesting the critical section to call kmalloc from this context, which is why I never ran into the scheduling while atomic issue before.
I'm fairly certain the critical section executes in thread context not interrupt context from either entry point.
I'm certain that the spinlock_t is only ever used in one function (a I posted a simplified version of the critical section earlier).
It seems that the critical section is often called in an atomic context. The spin_lock function sounds like it will only cause a second call to spin_lock to spin if it is called on a separate core.
But, since I'm certain the critical section is never called from interrupt context, only thread context, the fact that pre-emption is disabled on the core should provide the protection I need with out having to disable IRQs. Disabling IRQs would prevent an interrupt from occurring while the lock is acquired. I would like to avoid disabling interrupts if I don't need to.
So it sounds like spin_lock/spin_unlock is the correct choice?
In addition, I'd like to be more confident in my assumptions above. Can I test for atomic context? For instance, I know that you can call irqs_disabled(), is there a similar is_atomic() function I can call? I would like to put a few calls in different places to learn what sort of context I'm.
-Kai Meyer
On 11/10/2011 12:19 PM, Jeff Haran wrote:
-----Original Message----- From: kernelnewbies- bounces+jharan=bytemobile.com@kernelnewbies.org [mailto:kernelnewbies- bounces+jharan=bytemobile.com@kernelnewbies.org] On Behalf Of Dave Hylands Sent: Thursday, November 10, 2011 11:07 AM To: Kai Meyer Cc: kernelnewbies@kernelnewbies.org Subject: Re: Spinlocks and interrupts
Hi Kai,
On Thu, Nov 10, 2011 at 10:14 AM, Kai Meyer<kai@gnukai.com> wrote:
I think I get it. I'm hitting the scheduling while atomic because I'm calling my function from a struct bio's endio function, which is probably running with a lock held somewhere else, and then my mutex sleeps, while the spin_lock functions do not sleep. Actually, just holding a lock doesn't create an atomic context. I believe on kernels with kernel pre-emption enabled the act of taking the lock disables pre-emption. If it didn't work this way you could end up taking the lock in one process context and while the lock was held get pre-empted. Then another process tries to take the lock and you dead lock.
Jeff Haran
Kai, you might want to try bottom posting. It is the standard on these lists. It makes it easier for others to follow the thread.
I know of no kernel call that you can make to test for current execution context. There are the in_irq(), in_interrupt() and in_softirq() macros in hardirq.h, but when I've looked at the code that implements them I've come to the conclusion that they sometimes will lie. in_softirq() returns non-zero if you are in a software IRQ. Fair enough. But based on my reading in the past it's looked to me like it will also return non-zero if you've disabled bottom halves from process context with say a call to spin_lock_bh().
It would be nice if there were some way of asking the kernel what context you are in, for debugging if for no other reason, but if it's there I haven't found it.
I'd love to be proven wrong here, BTW. If others know better, please enlighten me.
Jeff Haran
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
I try to remember to bottom post on message lists, but obviously I've been negligent :)
Perhaps I'll just add some calls to msleep() at various places to help me identify when portions of my code are in an atomic context, just to help me learn what's going on.
-Kai Meyer
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
On 11/10/2011 11:58 PM, Rajat Sharma wrote:
For most of the block drivers bio_endio runs in a context of its tasklet, so it is indeed atomic context.
-Rajat
On Fri, Nov 11, 2011 at 4:50 AM, Kai Meyer<kai@gnukai.com> wrote:
On 11/10/2011 04:00 PM, Jeff Haran wrote:
-----Original Message----- From: kernelnewbies-bounces@kernelnewbies.org [mailto:kernelnewbies- bounces@kernelnewbies.org] On Behalf Of Kai Meyer Sent: Thursday, November 10, 2011 1:55 PM To: kernelnewbies@kernelnewbies.org Subject: Re: Spinlocks and interrupts
Alright, to summarize, for my benefit mostly,
I'm writing a block device driver, which has 2 entry points into my code that will reach this critical section. It's either the make request function for the block device, or the resulting bio->bi_end_io function. I do some waiting with msleep() (for now) from the make request function entry point, so I'm confident that entry point is not in an atomic context. I also only end up requesting the critical section to call kmalloc from this context, which is why I never ran into the scheduling while atomic issue before.
I'm fairly certain the critical section executes in thread context not interrupt context from either entry point.
I'm certain that the spinlock_t is only ever used in one function (a I posted a simplified version of the critical section earlier).
It seems that the critical section is often called in an atomic context. The spin_lock function sounds like it will only cause a second call to spin_lock to spin if it is called on a separate core.
But, since I'm certain the critical section is never called from interrupt context, only thread context, the fact that pre-emption is disabled on the core should provide the protection I need with out having to disable IRQs. Disabling IRQs would prevent an interrupt from occurring while the lock is acquired. I would like to avoid disabling interrupts if I don't need to.
So it sounds like spin_lock/spin_unlock is the correct choice?
In addition, I'd like to be more confident in my assumptions above. Can I test for atomic context? For instance, I know that you can call irqs_disabled(), is there a similar is_atomic() function I can call? I would like to put a few calls in different places to learn what sort of context I'm.
-Kai Meyer
On 11/10/2011 12:19 PM, Jeff Haran wrote:
-----Original Message----- From: kernelnewbies- bounces+jharan=bytemobile.com@kernelnewbies.org [mailto:kernelnewbies- bounces+jharan=bytemobile.com@kernelnewbies.org] On Behalf Of Dave Hylands Sent: Thursday, November 10, 2011 11:07 AM To: Kai Meyer Cc: kernelnewbies@kernelnewbies.org Subject: Re: Spinlocks and interrupts
Hi Kai,
On Thu, Nov 10, 2011 at 10:14 AM, Kai Meyer<kai@gnukai.com> wrote:
I think I get it. I'm hitting the scheduling while atomic because I'm calling my function from a struct bio's endio function, which is probably running with a lock held somewhere else, and then my mutex sleeps, while the spin_lock functions do not sleep. Actually, just holding a lock doesn't create an atomic context. I believe on kernels with kernel pre-emption enabled the act of taking the lock disables pre-emption. If it didn't work this way you could end up taking the lock in one process context and while the lock was held get pre-empted. Then another process tries to take the lock and you dead lock.
Jeff Haran
Kai, you might want to try bottom posting. It is the standard on these lists. It makes it easier for others to follow the thread.
I know of no kernel call that you can make to test for current execution context. There are the in_irq(), in_interrupt() and in_softirq() macros in hardirq.h, but when I've looked at the code that implements them I've come to the conclusion that they sometimes will lie. in_softirq() returns non-zero if you are in a software IRQ. Fair enough. But based on my reading in the past it's looked to me like it will also return non-zero if you've disabled bottom halves from process context with say a call to spin_lock_bh().
It would be nice if there were some way of asking the kernel what context you are in, for debugging if for no other reason, but if it's there I haven't found it.
I'd love to be proven wrong here, BTW. If others know better, please enlighten me.
Jeff Haran
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies I try to remember to bottom post on message lists, but obviously I've been negligent :)
Perhaps I'll just add some calls to msleep() at various places to help me identify when portions of my code are in an atomic context, just to help me learn what's going on.
-Kai Meyer
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Thanks so much for this Rajat. It helps a lot.
On Thu, Nov 10, 2011 at 3:10 AM, Dave Hylands <dhylands@gmail.com> wrote:
Hi Kai,
On Wed, Nov 9, 2011 at 1:07 PM, Kai Meyer <kai@gnukai.com> wrote:
When I readup on spinlocks, it seems like I need to choose between disabling interrupts and not. If a spinlock_t is never used during an interrupt, am I safe to leave interrupts enabled while I hold the lock? (Same question for read/write locks if it is different.)
So the intention behind using a spinlock is to provide mutual exclusion.
A spinlock by itself only really provides mutual exclusion between 2 cores, and not within the same core. To provide the mutual exclusion within the same core, you need to disable interrupts.
Normally, you would disable interrupts and acquire the spinlock to guarantee that mutual exclusion, and the only reason you would normally use the spinlock without disabling interrupts is when you know that interrupts are already disabled.
The danger of acquiring a spinlock with interrupts enabled is that if another interrupt fired (or the same interrupt fired again) and it tried to acquire the same spinlock, then you could have deadlock.
If no interrupts touch the spinlock, then you're probably using the wrong mutual exclusion mechanism. spinlocks are really intended to provide mutual exclsion between interrupt context and non-interrupt context.
Also remember, that on a non-SMP (aka UP) build, spinlocks become no-ops (except when certain debug checking code is enabled).
-- Dave Hylands Shuswap, BC, Canada http://www.davehylands.com
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Nice explanation Dave. Regards, Rohan Puri
Hi Dave,
Also, remember that spin-locks are no-ops on a single processor machine, so as coded, you have no protection on a single-processor machine if you're calling from thread context.
Not completely true, spinlock can still provide protection even in this case i.e. two thread context sharing a resource on UP. Remember that for peemptive kernel, it disables preemption, so it is guranteed that scheduler does not through out this thread as long as spinlock is held. I agree that a mutex can as well be used for this example but it again depends on situation, if you want to be quick about your task and are 100% sure that task does not sleep or takes too long processing cycles, disabling preemption (i.e. spinlock in this case) can be a better option. Thanks, Rajat On Thu, Nov 10, 2011 at 10:17 AM, rohan puri <rohan.puri15@gmail.com> wrote:
On Thu, Nov 10, 2011 at 3:10 AM, Dave Hylands <dhylands@gmail.com> wrote:
Hi Kai,
On Wed, Nov 9, 2011 at 1:07 PM, Kai Meyer <kai@gnukai.com> wrote:
When I readup on spinlocks, it seems like I need to choose between disabling interrupts and not. If a spinlock_t is never used during an interrupt, am I safe to leave interrupts enabled while I hold the lock? (Same question for read/write locks if it is different.)
So the intention behind using a spinlock is to provide mutual exclusion.
A spinlock by itself only really provides mutual exclusion between 2 cores, and not within the same core. To provide the mutual exclusion within the same core, you need to disable interrupts.
Normally, you would disable interrupts and acquire the spinlock to guarantee that mutual exclusion, and the only reason you would normally use the spinlock without disabling interrupts is when you know that interrupts are already disabled.
The danger of acquiring a spinlock with interrupts enabled is that if another interrupt fired (or the same interrupt fired again) and it tried to acquire the same spinlock, then you could have deadlock.
If no interrupts touch the spinlock, then you're probably using the wrong mutual exclusion mechanism. spinlocks are really intended to provide mutual exclsion between interrupt context and non-interrupt context.
Also remember, that on a non-SMP (aka UP) build, spinlocks become no-ops (except when certain debug checking code is enabled).
-- Dave Hylands Shuswap, BC, Canada http://www.davehylands.com
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Nice explanation Dave.
Regards, Rohan Puri
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
hi, Yes if config CONFIG_PREEMPT enabled,even if you dont have an smp box,we can think of preemption as equivalent to SMP.correct me if i am wrong about below lockings if we require locking only in user context: semaphore is preferred locking between user context(can be kernelpath) and sofirqs:spin_lock_bh() and spin_unlock_bh() locking between user context and tasklets:spin_lock_bh() and spin_unlock_bh() locking between user context and timers:spin_lock_bh() and spin_unlock_bh() locking between two same tasklets or same timers: not required locking between two different tasklets or different timers or tasklet/timer: spin_lock,spin_unlock locking between same softirqs: spin_lock,spin_unlock locking between different softirqs: spin_lock,spin_unlock locking between hardirq and softirq: spin_lock_irqsave,spin_unlock_irqrestore locking between two hardirq : spin_lock_irqsave thanks, karunakar On Thu, Nov 10, 2011 at 11:17 AM, Rajat Sharma <fs.rajat@gmail.com> wrote:
Hi Dave,
Also, remember that spin-locks are no-ops on a single processor machine, so as coded, you have no protection on a single-processor machine if you're calling from thread context.
Not completely true, spinlock can still provide protection even in this case i.e. two thread context sharing a resource on UP. Remember that for peemptive kernel, it disables preemption, so it is guranteed that scheduler does not through out this thread as long as spinlock is held.
I agree that a mutex can as well be used for this example but it again depends on situation, if you want to be quick about your task and are 100% sure that task does not sleep or takes too long processing cycles, disabling preemption (i.e. spinlock in this case) can be a better option.
Thanks, Rajat
On Thu, Nov 10, 2011 at 10:17 AM, rohan puri <rohan.puri15@gmail.com> wrote:
On Thu, Nov 10, 2011 at 3:10 AM, Dave Hylands <dhylands@gmail.com> wrote:
Hi Kai,
On Wed, Nov 9, 2011 at 1:07 PM, Kai Meyer <kai@gnukai.com> wrote:
When I readup on spinlocks, it seems like I need to choose between disabling interrupts and not. If a spinlock_t is never used during an interrupt, am I safe to leave interrupts enabled while I hold the lock? (Same question for read/write locks if it is different.)
So the intention behind using a spinlock is to provide mutual exclusion.
A spinlock by itself only really provides mutual exclusion between 2 cores, and not within the same core. To provide the mutual exclusion within the same core, you need to disable interrupts.
Normally, you would disable interrupts and acquire the spinlock to guarantee that mutual exclusion, and the only reason you would normally use the spinlock without disabling interrupts is when you know that interrupts are already disabled.
The danger of acquiring a spinlock with interrupts enabled is that if another interrupt fired (or the same interrupt fired again) and it tried to acquire the same spinlock, then you could have deadlock.
If no interrupts touch the spinlock, then you're probably using the wrong mutual exclusion mechanism. spinlocks are really intended to provide mutual exclsion between interrupt context and non-interrupt context.
Also remember, that on a non-SMP (aka UP) build, spinlocks become no-ops (except when certain debug checking code is enabled).
-- Dave Hylands Shuswap, BC, Canada http://www.davehylands.com
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Nice explanation Dave.
Regards, Rohan Puri
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
participants (6)
-
Dave Hylands -
Jeff Haran -
Kai Meyer -
Rajat Sharma -
rohan puri -
santhosh kumars