DjangoBot: #celery

and being local, it would be pretty pointless :)

could maybe just write our own

with MemcachedCache as base

cmemcache is pretty outdated

would be nice to support pylibmc

Shouldn't be too hard to translate

it has support for the binary protocol, compression, async, tcp nodelay etc

Well we could do a direct copy of theirs and switch it to use pylibmc for now, then come back to make it use the better features of pylibmc

Is the caching going to want to be django-based if using the django backend and this other caching we are talking about or just single caching setup and drop django cache support

not sure, that's something we should ask the mailinglist

but don't have to map out everything

wouldn't be that hard to do either way

I was thinking the loaders could have a override_backend attribute

so the django backend sets the "database" alias to be "djangocelery.backends.database" (or whatever it's called0

for backward compatibility

s/django backend/django loader

but not sure, need to see it in context when we have the sqlalchemy backend ready

if you want to use sqlalchmey and django, you could just type out the full class name

Well I am just going to replace all the signalling stuff with blinker and use that time to get associated with the code bad

base

Only have a few more hours of dev time today before other commitments take me away from it

the general is putting me in charge of washing clothes here now, been promoted ;)

Oh nice!

Sometime over the next week or two, it'd be nice to get this mapped out so we have set goals to meet

Even if it is a loose map, a good idea of where everything is going makes it easier to hack things out

agreed

good to you have you on board!

:)

apollo13: I wont be working on the sqlalchemy stuff this weekend, just the signal stuff

I have a test celery project setup with celery and MQ

i dont get what the role of MQ is

ed1t: MQ as in what?

RabbitMQ

i have rabbitMQ, celery and django app running

ah. so the questions is, why do you need rabbitmq?

yes

when you run a task, it gets sent to rabbit, which sends it of to one of possily several workers. One of the workers finish the task and sends it back as another message via rabbit

ok so what does the CELERY_BACKEND is used for?

that deals with how the results of a task is stored

(see also http://celeryproject.org/docs/configuration.html )

i did that part i have configured mogodb

but i dont see anything written to mongodb

but you can run tasks and query the tasks about their status?

nmv i see it now, i had to do syncdb

it only writes Task and not PeriodicTask? I ran 2 tasks and have 1 periodictask running

i only see 2 entries in my mongodb collection

Presumably you will get one entry every time the periodic task runs

i dont see anything

I'm not too well versed in the innards of the task data storage though

this is what I see: { "_id" : "8410fb56-c984-4b7e-8e8a-fb89a958028b", "status" : "SUCCESS", "result" : BinData type: 2 len: 9, "date_done" : "Sun Mar 21 2010 14:02:57 GMT-0400 (EDT)", "traceback" : BinData type: 2 len: 6 }

which doesnt really make sense...it doesnt say which task

the id is there

which is what you use to identify an instance of a task

but normally you just call methods on the task object. (see http://celeryproject.org/docs/getting-started/f... )

backend is optional right?

which backend?

the backend which stores the result of tasks

in my case, mongodb

No, it's not optional as far as I know. Then you would not be able to get results back from your tasks

but you can use amqp as the backend, so you wont need a separate db or mongo or anything. there are some caveats. see the config doc I pasted above

what I was going to do was when the task is ran, in that task I was going to catch the error

as in, inside of your task you want to catch an expection or something?

is's probably easier if you can paste some code on a pastebin or something

well im just using one of the sample apps, http://pastebin.com/GHWJdRm5

lets say there was any error while processing it, i will to try: except:

all the datastore is going to tell me is that whether it was succesful or not, it wont tell me the errors

you can get the exception back as the result of the task

it will be serialized and sent back, just as the result of the task would

you can also handle the exception inside of the task if there is a chance you can recover from it

You can also refire the task from inside of the task using retries

so if it fails it will try again

good for when making external connections

if mytask.status == "FAILURE" then mytask.result contains the exception

and yeah. That's a normal case. retrying whenever you get a failwhale exception or something

I'm wondering if the docs needs some prose about catching exceptions though. the api docs are a bit daunting for finding out about something like error handling

where do I call mytask.status from?

ed1t: when you call your task you get a task object back, on which you can check status and so on

see http://celeryproject.org/docs/getting-started/f...

ed1t: that docs page says "The AsyncResult lets us find the state of the task, wait for the task to finish and get its return value (or exception if the task failed)."

in django, if i make it wait for the task wouldnt that kinda hang the page too

yeah. so in that case, you shouldn't wait for it

as you're right, that would defeat he whole point of using celer :)

yea

can't you just look it up by it's id

so is there like api method where I can do like get_task and pass the id was generated

ed1t: you should be able to do taskresult = celery.result.AsyncResult(taskid)

as in. in your view or whatever. taskid = MyTask.delay(); save the taskid somewhere and pass it to the AsyncResult constructor later on

I guess ask would be the guy to ask if that's the right way, but he's doing laundry and was hit with a netsplit

erh. retval from delay is a result object with a task_id member. so that's the one to save

yep