DjangoBot: #celery

how do u delete a task in celery?

ed1t: what I've done is have the task check if it's been 'cancelled' against some system (in my case a mysql table)

and if so it creates just returns and doesn't so the work

there's also celery.task.control.revoke(task_id)

Hi Everyone

my celery gets unusable. I explain:

I have tasks which emit more tasks (which subsequently emit even more tasks).

They are collecting data and updating a database.

I don't expect any results/output from these tasks. So, fire-and-forget

In approx. 20-24 hours it just gets stalled. 100% cpu, zombie python celeryd processes.

Should I remember all the task IDs, store them somewhere and revoke them later on?

I want them disappear forever after job is done (or failed).

How do I auto-cleanup?

Does not matter which result backend I use.... always the same... v1.0.1.

Hi. I'm running celeryd with manage.py in Django and I'm currently developing, so I need to restart celeryd occassionally. I was wondering what the best way to kill the service is. I read online that there should be a *.pid file that is created in the directory that holds the pid I need to kill the process but I do not get such a file. Any ideas?

There is a long delay (20-30 seconds) between the time Celery picks up a job and when it starts running it. How do I remove this delay?

that doesn't sound right

should take far less than a second

does it have an eta?

asksol: eta?

you can set the time for when after the task should execute

countdown/eta

shouldn't, using defaults

rabbitmq?

yup

that's not normal at all

:(

ideas?

it doesn't say anything about eta in the log message?

nope

i'll go double check

try the demoproject in examples/python

[2010-03-31 18:36:12,745: INFO/MainProcess] Got task from broker: xxxx[a99bc317-7bd4-4cff-955f-672d3d4271d1]

[2010-03-31 18:37:10,629: INFO/MainProcess] Task xxxx[a99bc317-7bd4-4cff-955f-672d3d4271d1] processed: [....

can't easily run the demo project

hi guys

anyone knows how to control which user celery runs at?

the debian init.d suggests a var CELERYD_USER

but should that be a env var or one in settings.py?

perhaps the init scripts?

yes that's a straight forward place for it

but is that the accepted way

?

i don't know; new to celery myself, but I suspect that it is since that init script and general user control is spesfic to debian

if i set CELERYD_USER="user" it won't start

you created a user called user?

gpmidi_wrk: try to run with --loglevel=DEBUG

trying...

then you can see how fast it's sent to the pool

maybe the task itself is slow

or the pool is the culprit

the job should take 5s max

it's a very quick sql query

do you think dns might be the culprit?

it has to connect to the database for every task

hmm

unless you set CELERY_DB_REUSE_MAX

to rabbit you mean?

(in 1.0.2)

to the database

oh, hmm, i don't think it has any tables in the db

gpmidi_wrk, no

it was for aggmment

harel_: oh, you just repalced the username when you pasted it?

yes

harel_: can you su up to that user?

harel_: add it to /etc/default/celeryd

asksol, any ideas about CELERYD_USER?

yes

i can

ok

thanks

or; export CELERYD_USER=foo

the hostname is staticly set in /etc/hosts

gpmidi_wrk: is celery.ping processed fast?

trying

what module is ping in?

asksol, not working.... nothing starts

i used quotes. should drop it maybe

oh, celery.task

hmm. no..

no, ping is slow too

harel_: with another user, you need to set --schedule

e.g. --schedule=/home/users/celerybeat-schedule

i'd almost suspect a time issue but it's all on the same host...

e.g. CELERYD_OPTS="--schedule=/home/users/celerybeat-schedule"

gpmidi_wrk: did you run with --debug?

yup

asksol, i'm referring to the celeryd process

harel_: ah, ok. so you're not using beat?

oh hey, i'm getting something in the logs:

"TaskPool: Finding dead pool processes..." right after the task runs

what do you mean it doesn't start?

i am, but i hvaent got to that problem yet ;o)

gpmidi_wrk: that's expected, it just supervises the pool

right now, at 3am my concern is celeryd running as my right user