DjangoBot: #celery

asksol: you there?

curious strikes: if on .8, we're running some periodic tasks, and we don't notice for a few days that, oh id on't know, celerybeat isn't running. would those tasks ever get run in any fashion?

Hi, I am playing with celery. Just wondering if there is anyway of pausing a task so that you may come back to it at a later time?

avisual, You can set an eta so a task runs at a specific time (datetime instance)

avisual: I'm curious... why would you want to do that?

avisual, or you can use countdown (i think) so set a task to run in X seconds

avisual: you could add a call to sleep() in your task, but I'm not sure why you would want to do that

if i have a number of tasks that are doing bulk inserts into DB. my limiting factor is I/O if i want to stop/pause a task so that i could shift another task into place as that new task became a higher priority

Did that make sense?

avisual: I don't think there's any direct support in celery to do something like that, but you could certainly implement it yourself by having the task monitor a message queue and reschedule itself for later

celery does support priorities, but I don't think it will pause a task for you

ok so you can reschedule? Thanks :)

oh... I was wrong... AMQP supports priorities, but RabbitMQ doesn't implement them

Ahh but if the task is running already. I'll need to record where it is in its task and then stop it and add it back into the que

Mulka : thanks

ah, finally some real coffee

avisual: you're implementing distributed coroutines? :)

or continuation

that would be awesome, if you could send the stack of a generator and continue it elsewhere

I wonder if that's how Seaside et.al does it?

etc

Hi all - quick question. Say I have a Task with a high eta (like, tomorrow) - how do I cancel it?

celery.task.control.revoke(task_id)

but currently, this does not persist

so if you restart the workers, the revokes are lost

but that's probably fine for this

oh, there's a bug there actually

that's a use case I didn't think through

if the task is already received, and is hold by the scheduler, it won't be revoked until the eta has passed

hmm, how much work would you estimate it would be to persist revokes, and cancel them as soon as revoke is called?

I'm happy to help with this

persist revokes, I've been thinking about it

celerybeat already uses shelve to store persistent data

celeryd should probably also have a shelve to store things in

makes sense

it probably shouldn't take that long

i'm fixing the schedule bug thing right now

also, on a somewhat related note

cool, thanks!

I have a mixed environment - one part Django and one part Twisted

Celery fits in brilliantly with the Django part (thanks for this)

the only thing I'm concerned with (and keep in mind I haven't really read any celery code yet)

is executing task.delay within the Twisted reactor

task.delay shouldn't be blocking, right?

that is, it must simply send a message to the broker and return a id to track the pending task

I image it does this in a thread?

imagine

rlotun, you are correct sir.

it is blocking, in the sense it opens up a connection and sends the message

for now, I can run task.delay in a thread within Twisted then to make it "non-blocking"

yeah

I wonder though, a Twisted-safe task could be useful

maybe you could add support for sending it via txAMQP as well

yes, I was just thinking that

I'll have a play with that, and let you know

any pointers in general? I was thinking of writing some extra task.control routines to support this

make sense?

to support what?

Twisted-safe task submissions using txAMQP

the most complete way would be to implement a carrot backend for it

it already has a Pika asyncore backend

ahh, ok

but I'm the only one that has ever written a carrot backend, so :)

who

but it's just sending a message

.control is for broadcast control command messages

ok, got you

just looking at the carrot backends in github now

Asksol: Do you think it could be done re. continuation/coroutines?

maybe in a functional language, not so sure with python

at least not easily, maybe with some C or bytecode

rlotun: http://github.com/ask/celery/commit/cfbeeecbe6b...

avisual: Maybe with Stackless or greenlets

asksol: cool, thanks

yeah, stackless or greenlets, good idea

or at least good place to look. it would be an awesome feature

hanks will have a look at them

I'll keep the room updated with what i find

although, probably easier to get what you want by manual means

asksol: regarding Twisted, everything boils down to TaskPublisher.delay_task, which sends off a Django signal, right? what listens on that signal?

nothing by default

oh ok, I see now

It's not recommended to listen to it either

ahh, sorry, missed out that it's the Publisher from carrot.messaging, and the send method. Ok, I've followed it all the way through and understand it better now ;-)

i have it in my head that celery has a rest api - am i crazy?

and of course a quick trip to the docs answered that.

Hello, I am writing a simple postgresql custom job management web app, and they pointed me to celery

is there a front end that shows me what celery jobs have been started - are finished?

you can try setting up http://github.com/ask/celerymon

ah

I'll take a look at that

how do I ask the registry if my task is done?

actually , let me give more context... first I do a request using delay, and the process starts in the background... then, I want to implement a function that asks the registry if it is done

rburhum: result = Task.delay(...)

result.is_ready()

but I don't have the result

it is on a different request

sorry.