2010-2-4
| 04:28 | yashh | hey |
| 04:41 | mulka | any idea how to do subtasks correctly? |
| 05:52 | yashh | hello |
| 07:01 | asksol | yashh |
| 07:01 | asksol | hey |
| 07:01 | yashh | hey |
| 07:01 | yashh | how are you |
| 07:02 | asksol | great |
| 07:02 | asksol | tired |
| 07:02 | asksol | you? |
| 07:02 | yashh | i m good |
| 07:02 | yashh | just changing jobs |
| 07:02 | yashh | busy |
| 07:02 | yashh | few qns on celery |
| 07:03 | yashh | I am trying to use celery with redis |
| 07:03 | yashh | the docs say to have CELERY_BACKEND as pyredis |
| 07:03 | yashh | but the dictionary in backends/__init__ seems to have "redis" as key |
| 07:12 | asksol | ah |
| 07:12 | asksol | thanks |
| 07:12 | asksol | that's a typo then |
| 07:12 | yashh | also do we BROKER settings when using redis |
| 07:13 | asksol | well, this is all a bit confusing, but backend in celery is actually the backend used to store results |
| 07:13 | asksol | the backend used to send messages is CARROT_BACKEND |
| 07:14 | yashh | ohh ok |
| 07:14 | asksol | |
| 07:14 | yashh | cool github is back. Its been down for a long time |
| 07:16 | yashh | asksol: Is rabbitMQ + database the most recommended setting for celery ? |
| 07:20 | asksol | definitely |
| 07:20 | asksol | oh |
| 07:21 | asksol | not database |
| 07:21 | asksol | for storing the results, cache or riak |
| 07:21 | asksol | urm, redis |
| 07:21 | asksol | sorry. I was just reading the riak news :) |
| 07:22 | yashh | it support map reduce via javascript lol |
| 07:22 | asksol | so rabbitmq + cache/redis/mongodb/anything except database |
| 07:22 | yashh | cool |
| 07:22 | yashh | so BROKER = rabbitMQ |
| 07:22 | asksol | but you could always just skip saving results altogether |
| 07:23 | yashh | yeah I dont think I have seen results saving |
| 07:23 | asksol | the plan in the future is to use amqp to send back the result, and use a result backend to cache the result |
| 07:31 | yashh | asksol: If you have a rabbitmq.conf file can you share. its working fine with default settings but just wanted to know if there is any fine tuned settings file |
| 07:33 | asksol | I just use the defaults actually |
| 07:37 | asksol | i'm not sure what you can set there even, NODENAME, and high_memory_watermark at least |
| 07:38 | yashh | :) yup I always go with default guest:guest |
| 07:38 | yashh | Is celerymon good to install ? |
| 07:42 | yashh | almost 12PM here good night |
| 08:22 | asksol | nice. tests are passing when using django trunk |
| 10:19 | asksol | celery-users is now the second hit for "celery" on google |
| 18:05 | davidcramer | asksol: have you really tested celery much with transactions in django mysql innodbs? |
| 18:07 | davidcramer | then I suppose, since the API looks similar, is a Task just a Thread? :) |
| 18:09 | davidcramer | then even further, if you had to aggregate 1 million feeds, would you just dump a task for every feed? |
| 18:10 | davidcramer | my current approach was to run a periodic check for feeds which needed sync'd, and then run a bunch of threads to work on those |
| 18:10 | davidcramer | so I suppose I could do the same, periodic task + sending a bunch of tasks |
| 18:48 | davidcramer | And I suppose, my last question, do workers need strings/whatever passed as args, or can objects work ? |
| 19:36 | mulka | is there a way to have a single server work on multiple queues? |
| 21:19 | asksol | |
| 21:20 | asksol | davidcramer: we do exactly that |
| 21:20 | asksol | or, there's some other optimizations involved |
| 21:20 | asksol | like it applies n tasks at a time |
| 21:21 | asksol | I can send you some code |
| 21:21 | asksol | it should be open sourced soon |
| 21:21 | asksol | bbl |
| 22:13 | d0ugal | umm what us the best/recommended way to have celeryd start when the server boots? (Ubuntu) |
| 22:24 | mulka | d0ugal: not sure if its the best way, but I would start it using start-stop-daemon in /etc/rc.local |
| 22:48 | mulka | how does a celery worker decide which queue to pull from if there are multiple? Does each "process" just round robin or something? |
| 22:52 | asksol | |
| 22:52 | asksol | mulka: rabbitmq decides |
| 22:52 | asksol | bbl |
| 22:57 | mulka | asksol: but if a worker is assigned to multiple queues... |
| 22:57 | mulka | its not like each worker has its own queue to pull from, right? |
| 23:54 | davidcramer | asksol: just a rough draft based on some of your recipes :) http://www.pastethat.com/AkxJX |