#celery
Feb. 4th, 2010
| 07:33 | asksol | I just use the defaults actually |
| 07:37 | asksol | i'm not sure what you can set there even, NODENAME, and high_memory_watermark at least |
| 07:38 | yashh | :) yup I always go with default guest:guest |
| 07:38 | yashh | Is celerymon good to install ? |
| 07:42 | yashh | almost 12PM here good night |
| 08:22 | asksol | nice. tests are passing when using django trunk |
| 10:19 | asksol | celery-users is now the second hit for "celery" on google |
| 18:05 | davidcramer | asksol: have you really tested celery much with transactions in django mysql innodbs? |
| 18:07 | davidcramer | then I suppose, since the API looks similar, is a Task just a Thread? :) |
| 18:09 | davidcramer | then even further, if you had to aggregate 1 million feeds, would you just dump a task for every feed? |
| 18:10 | davidcramer | my current approach was to run a periodic check for feeds which needed sync'd, and then run a bunch of threads to work on those |
| 18:10 | davidcramer | so I suppose I could do the same, periodic task + sending a bunch of tasks |
| 18:48 | davidcramer | And I suppose, my last question, do workers need strings/whatever passed as args, or can objects work ? |
| 19:36 | mulka | is there a way to have a single server work on multiple queues? |
| 21:19 | asksol | |
| 21:20 | asksol | davidcramer: we do exactly that |
| 21:20 | asksol | or, there's some other optimizations involved |
| 21:20 | asksol | like it applies n tasks at a time |
| 21:21 | asksol | I can send you some code |
| 21:21 | asksol | it should be open sourced soon |
| 21:21 | asksol | bbl |
| 22:13 | d0ugal | umm what us the best/recommended way to have celeryd start when the server boots? (Ubuntu) |
| 22:24 | mulka | d0ugal: not sure if its the best way, but I would start it using start-stop-daemon in /etc/rc.local |
| 22:48 | mulka | how does a celery worker decide which queue to pull from if there are multiple? Does each "process" just round robin or something? |
| 22:52 | asksol | |
| 22:52 | asksol | mulka: rabbitmq decides |
| 22:52 | asksol | bbl |
| 22:57 | mulka | asksol: but if a worker is assigned to multiple queues... |
| 22:57 | mulka | its not like each worker has its own queue to pull from, right? |
| 23:54 | davidcramer | asksol: just a rough draft based on some of your recipes :) http://www.pastethat.com/AkxJX |
Feb. 5th, 2010
| 00:10 | d0ugal | asksol, FYI there seems to be a typo here; http://celeryproject.org/tutorials/otherqueues.... I think toproot should be taproot? |
| 03:51 | yashh | hey |
| 03:54 | mulka | yashh: hi |
| 03:54 | yashh | I see this in my celery logs |
| 03:54 | yashh | AMQP Listener: Connection Error: [Errno 111] |
| 03:56 | mulka | may celery is having trouble connecting to your AMQP server? have you double checked the settings? |
| 03:56 | mulka | is celery actually working, or not? |
| 04:12 | yashh | mulka: sorry, yes celery is working |
| 04:12 | yashh | but once in a while I see that in my log |
| 04:16 | mulka | yashh: I don't really have a better answer for you. someone else might know more |
| 04:16 | yashh | hey np. I dont think its major but just something. |
| 10:00 | asksol | d0ugal: yeah, thanks! already fixed in the latest version btw |
| 10:00 | asksol | development docs that is |
| 12:58 | asksol | released celery 0.8.4 |
| 12:58 | asksol | just a small maintenance release |
| 12:58 | asksol | it now pukes out a warning if --detach is used, and some typos fixed |
| 17:58 | madness | is there an easy way to get at the number of tasks left in a queue via the celery or carrot apis ? It wasn't clear to me in the docs.. |
| 18:33 | asksol | madness: well, not programatically |
| 18:33 | asksol | but it will be easier with celerymon |
| 18:33 | asksol | rabbitmqctl list_queues -p myvhost |
| 18:34 | madness | asksol: yup, I figure I could just parse that out. I have a task that spits more tasks into the queue for a crawler once every now and then, but don't want it to continue doing so when the queue is bigger than 50k |
Feb. 6th, 2010
| 03:18 | abadr | When I use the Supervised flag w/mange.py celeryd, it doesn't process any tasks and I think the workers keep dying |
| 04:06 | abadr | I guess I'll just run it without -S for now... |
| 18:59 | davidcramer | I think this is asked 12 times a day, but is there an easy way to see current queued tasks? |
Feb. 7th, 2010
| 05:43 | davidcramer | asksol: amqp as a tombstone backend, that warning, i wanted to confirm what "receive the result once" means :) |
| 05:46 | davidcramer | i set the concurrency to 4, and it shows 5 procs, kinda of odd ;) |
| 05:48 | davidcramer | yay, celery running on demo site :D |
| 05:48 | davidcramer | |
| 06:02 | davidcramer | ooo, error emails :D |
| 06:03 | davidcramer | and i suppose i should fork/commit this back, but for now ill just post here: http://celeryproject.org/reference/celery.task.... |
| 06:03 | davidcramer | kwargs/args should really be optional I think |
| 06:46 | davidcramer | in fact i will probably fork tomorrow |
| 06:46 | davidcramer | i think i have some useful additions to commit back |
| 09:05 | asksol | davidcramer: there's no easy way to see queued tasks until celerymon is done |
| 09:06 | asksol | 5 processes, one main process and 4 pool workers? |
| 09:06 | asksol | you can see what they do in ps listings if you install the "setproctitle" module |
| 09:07 | asksol | |
| 09:08 | asksol | thanks! |
| 09:08 | asksol | amqp backend means you send the result as a message, so you can only consume that result once |
| 09:09 | asksol | so if you do result.get() in two different processes, you will have a deadlock |
| 09:09 | asksol | or deadlock is probably not the right term |
| 09:09 | asksol | plan for 1.2 is to have an option to cache the result in a result backend |
| 09:10 | asksol | so send back result with amqp, e.g. store it in redis when consumed |
| 18:59 | davidcramer | I'm loving Celery :D |
| 18:59 | davidcramer | we've gotten it all moved into tasks and its working amazing |
| 18:59 | davidcramer | so any heavy task no longer happens on the web |
| 18:59 | davidcramer | should make scaling a lot easier as well |
| 19:00 | davidcramer | I'm assuming I could just have a project+celery install on any server, then one rabbitmq install for it to communicate with? Easy scaling, ya? |
| 19:00 | asksol | :) |
| 19:00 | asksol | indeed |
| 19:01 | davidcramer | cool |
| 19:01 | davidcramer | I did have one transactional issue |
| 19:01 | davidcramer | and I have had the same issue without celery, so I would say its more of a flaw in Django |
| 19:01 | davidcramer | |
| 19:02 | davidcramer | for some reason django doesnt commit the save() transaction here |
| 19:02 | davidcramer | (at the bottom) |
Feb. 8th, 2010
| 04:56 | gnperumal | hi |
| 04:57 | gnperumal | without rabbitq, how configure the celery and django |
| 04:57 | gnperumal | pls, its urgent |
| 05:34 | davidcramer | i thought rabbitmq was a requirement |
| 05:35 | davidcramer | or is that entirely based on the backend setting? |
| 06:12 | gnperumal | celery with databse is it possible |
| 07:43 | asksol | |
| 16:06 | regent | Hi |
| 16:06 | regent | I have a question regarding signals in celery |
| 16:07 | regent | is the signal 'task_postrun' send every time a request runs ? |
| 16:41 | asksol | request? |
| 16:41 | asksol | it's sent when after a task has been executed |
| 16:42 | asksol | executed by the worker, or using apply()/ALWAYS_EAGER locally (happens in celery.execute.trace) |
| 16:43 | asksol | but this is in 1.0, in 0.x.x, it was only sent when executed by the worker |