DjangoBot: #celery

still ponding how to do subtasks... haven't figured out an ideal solution yet. anyone want to brainstorm?

pondering*

anyone here?

mulka: subtasks?

yea... one task depends on a set of other tasks

asksol?

just launching them, or depends on their result?

depends on the result

well, that's when it gets funky

the best way is to use callbacks

callbacks all the way down

and never wait for the result

how would I do a callback with celery?

e.g. @task def refresh_feed(feed_url, callback=None): result = refresh_the_feed(); if callback: callback(result)

that result could be another task for example

s/result/callback

refresh_feed(url, callback=update_subscription_counts.delay)

siwu was even working on a task chain wrapper

didn't realize you could pass functions as arguments to tasks

well, you can, as long as you use pickle

does the code get passed across the wire, or does it just use the code that's on the worker server?

to be compatible with json as well, you could just pass the name of a task, and then in the code: from celery.registry import tasks; callback = tasks[callback]; callback.delay(result)

it doesn't serialize the actual source code

if you think about it, that would be kinda crazy :)

>>> pickle.dumps(operator.add)

'coperator\nadd\np0\n.'

serializing source code I think has been done before, but I don't think its straight forward to implement

that's what picloud does, right?

no, it doesn't actually

or, it does that for lambda

lambdas

and for stuff defined in __main__

but you still need to upload any modules to your vm

imagine disassembling most of django and assembling it on the other side again

that would be kind of silly

I'd like to steal the picloud pickler, but it's not open source

let's say I have a task that depends on the results of two other tasks. how would I get the results from the other two tasks together in the same function?

well, that's where it gets really tricky

using TaskSet and waiting for it is the easiest

but it's not optimal

read this thread: http://groups.google.com/group/celery-users/bro...

asksol: great work on Celery by the way! it's shaping up to be a really useful piece of software

thanks!

seems like a TaskSet itself should be able to have a callback passed to it

task chains could be interesting. you said siwu was working on something? anywhere I could get more info?

you'd have to ask him

it's not that easy to track when a taskset is done

but it was an idea yeah

to keep a counter, and the last task calls the callback when the counter hits the taskset size

padt: killall usually doesn't work for me

maybe if you write killall python manage.py celeryd, or something like that

asksol: ok. I just assumed it would do more or less the same thing as what's in the docs. If it doesn't work just kill the fixme

at least, that's the universal solution :)

works on an old sparcstation, works now

heh

asksol: is there anyway of setting a routing_key for httpdispatchtask

Oh... another question... what result store do you recommend for large results? Seems like a lot of them can't handle things bigger than 1MB.

mat__: HttpDispatchTask.apply_async(..., routing_key=)

mulka: what is large? for really large files maybe you should use HDFS :)

large results

that's for gigabyte large.

I think my results are only going to be a few MBs

asksol: HttpDispatchTask.apply_async(args=[TORNADO_URL, action_template, instance.follower_id], routing_key='activity.stream.follow') tried that, just having serious thinking issues about how to do kwargs

can you pass in a dict to kwargs?

yes, of course

no it's the special CeleryKeywordArgumentDictWriter

lol, too early for me

was up re deploying machines till 3 am, and started again at 9am

brain dead

hehe, i understand

just broke all our stream system, doh!

ouch

mulka: the best is to not send large results

but I guess you know that

just trying to make you think of ways to avoid it

asksol: thanks a lot again, back online now

still cant find a way of testing real time streams

mat__: real time streams?

I wrote some context thingy to test the twitter streamin api. might be what you need

this one. http://bitbucket.org/runeh/servercontext/src/

it the stream is http that is

redditors have no sense of humour: http://www.reddit.com/r/Python/comments/b0g9u/c...

hehe

nice nice!

I'm trying to run celeryd as a Windows service. Am I in uncharted waters here?

scratch that, I'm *thinking* of running celeryd as a Windows service, and trying to gauge how much work will be involved.

donspaulding: http://botland.oebfare.com/logger/celery/2010/0...