Freeing work_struct memory

[Kai Meyer]

I've got a bug I'm having trouble identifying. It seems like it could be 
related to my work_struct usage. I essentially have this:

struct my_worker {
         work_struct work;
         /* some other data */
};

void worker_fn(struct work_struct *work)
{
         struct my_worker *worker = container_of(work, struct my_worker, 
work);
         /* ... do some stuff ... */
         kfree(worker);
}

void worker_caller()
{
         struct my_worker *worker = kmalloc(sizeof(*worker), GFP_KERNEL);
         INIT_WORK(&worker->work, worker_fn);
         /* ... add some other stuff to *worker ... */
         schedule_work(&worker->work);
}

I frequently get a kernel panic with a specific test, but the stack 
trace is rarely the same, which seems to indicate to me that I'm 
corrupting data somewhere. So my question is:

Can I free the memory for "struct my_worker *worker" inside worker_fn? 
Or does the work_queue stuff need to continue to use the "struct 
work_struct work" member after the end of worker_fn?

-Kai Meyer