DjangoBot: #celery

hey

any idea how to do subtasks correctly?

hello

yashh

hey

hey

how are you

great

tired

you?

i m good

just changing jobs

busy

few qns on celery

I am trying to use celery with redis

the docs say to have CELERY_BACKEND as pyredis

but the dictionary in backends/__init__ seems to have "redis" as key

ah

thanks

that's a typo then

also do we BROKER settings when using redis

well, this is all a bit confusing, but backend in celery is actually the backend used to store results

the backend used to send messages is CARROT_BACKEND

ohh ok

cool github is back. Its been down for a long time

asksol: Is rabbitMQ + database the most recommended setting for celery ?

definitely

oh

not database

for storing the results, cache or riak

urm, redis

sorry. I was just reading the riak news :)

it support map reduce via javascript lol

so rabbitmq + cache/redis/mongodb/anything except database

cool

so BROKER = rabbitMQ

but you could always just skip saving results altogether

yeah I dont think I have seen results saving

the plan in the future is to use amqp to send back the result, and use a result backend to cache the result

asksol: If you have a rabbitmq.conf file can you share. its working fine with default settings but just wanted to know if there is any fine tuned settings file

I just use the defaults actually

i'm not sure what you can set there even, NODENAME, and high_memory_watermark at least

:) yup I always go with default guest:guest

Is celerymon good to install ?

almost 12PM here good night

nice. tests are passing when using django trunk

celery-users is now the second hit for "celery" on google

asksol: have you really tested celery much with transactions in django mysql innodbs?

then I suppose, since the API looks similar, is a Task just a Thread? :)

then even further, if you had to aggregate 1 million feeds, would you just dump a task for every feed?

my current approach was to run a periodic check for feeds which needed sync'd, and then run a bunch of threads to work on those

so I suppose I could do the same, periodic task + sending a bunch of tasks

And I suppose, my last question, do workers need strings/whatever passed as args, or can objects work ?

is there a way to have a single server work on multiple queues?

mulka: http://ask.github.com/celery/faq.html#can-i-sen...

davidcramer: we do exactly that

or, there's some other optimizations involved

like it applies n tasks at a time

I can send you some code

it should be open sourced soon

bbl

umm what us the best/recommended way to have celeryd start when the server boots? (Ubuntu)

d0ugal: not sure if its the best way, but I would start it using start-stop-daemon in /etc/rc.local

how does a celery worker decide which queue to pull from if there are multiple? Does each "process" just round robin or something?

d0ugal: http://github.com/ask/celery/tree/master/contri...

mulka: rabbitmq decides

bbl

asksol: but if a worker is assigned to multiple queues...

its not like each worker has its own queue to pull from, right?

asksol: just a rough draft based on some of your recipes :) http://www.pastethat.com/AkxJX