Poll for output in Minions
The main idea was to be able to get some output during the minion side execution. The idea maybe useful for tasks which are kind of long and user wants to get some notification about what is going on on the other side. Because we are talking about long running operations that stuff is reasonable only for async stuff. Here is some example of tracking a long operation on overlord :
[root@acerfedora func]# func "*" call --logpoll --async echo run_str_log "eeeeee" Logging data is initializing ... ------HOST : acerfedora ------- 2009-07-28 20:06:55,013 - 1248800814.958679-minion - INFO - Starting counting logger ... ------HOST : acerfedora ------- 2009-07-28 20:08:00,034 - 1248800814.958679-minion - INFO - Calling method with counter is 0 ------HOST : acerfedora ------- 2009-07-28 20:08:05,033 - 1248800814.958679-minion - INFO - Calling method with counter is 1 ------HOST : acerfedora ------- 2009-07-28 20:08:10,034 - 1248800814.958679-minion - INFO - Calling method with counter is 2 ------HOST : acerfedora ------- 2009-07-28 20:08:15,034 - 1248800814.958679-minion - INFO - Calling method with counter is 3
That example is for lots of machines (overlord spec) but polling output seems more reasonable for doing it for an one host :
[root@acerfedora func]# func "*" call --nopoll --async echo run_str_log "eeeeee" JOB_ID: '*-echo-run_str_log-1248802501.952774' [root@acerfedora func]# [root@acerfedora func]# func "*" call --logone="acerfedora" "*-echo-run_str_log-1248802501.952774" 2009-07-28 20:35:02,318 - 1248802502.256377-minion - INFO - Starting counting logger ... 2009-07-28 20:36:07,318 - 1248802502.256377-minion - INFO - Calling method with counter is 0 2009-07-28 20:36:12,317 - 1248802502.256377-minion - INFO - Calling method with counter is 1 2009-07-28 20:36:17,317 - 1248802502.256377-minion - INFO - Calling method with counter is 2 2009-07-28 20:36:22,317 - 1248802502.256377-minion - INFO - Calling method with counter is 3 2009-07-28 20:36:27,318 - 1248802502.256377-minion - INFO - Calling method with counter is 4 2009-07-28 20:36:32,318 - 1248802502.256377-minion - INFO - Calling method with counter is 5
Making a method to have trackable output
It is pretty easy let see it on an example a method in echo module :
def run_str_log(self,command):
import time
self.run_str_log.logger.info("Starting counting logger ...")
time.sleep(60)
for i in range(100):
time.sleep(5)
self.run_str_log.logger.info("Calling method with counter is %d"%i)
return command
By default every method in minion modules is attached with a standart logger instance.To reach the log instance what you need is calling self.method.logger and you have all the stuff in a standart logging.
Python API :
In [1]: from func.overlord.client import Overlord
In [2]: fc = Overlord("*",async=True)
In [3]: j=fc.echo.run_str_log("blip")
In [4]: j
Out[4]: '*-echo-run_str_log-1248802917.3343029'
In [5]: fc.tail_log(j)
Out[5]:
({'acerfedora': ['2009-07-28 20:41:57,572 - 1248802917.536685-minion - INFO - Starting counting logger ...\n',
'']},
False)
About how to make that code useful in your applications you should take a look into overlord/cmd_modules/call.py
Also you can poll a single machine if you need , which maybe useful in webapps or some other stuff.
In [6]: fc = Overlord("*",async=True)
In [7]: j=fc.echo.run_str_log("blip")
In [8]: fc.tail_log(j,host="acerfedora")
Out[8]:
({'acerfedora': ['2009-07-28 20:43:45,233 - 1248803025.1892841-minion - INFO - Starting counting logger ...\n',
'']},
False)
Under the hood :
Because we use xml-rpc as transport protocol these kind of things are not very efficient in func. But the best way to do that was to log all async requests in different log files according to their minion job ids. When a new async request comes a log object is attached to called method and new log file is created named with minion job id.Therefore user can use that log object to log stuff and overlord stuff can track that log file via its (overlord job_id)--(minion job_id) matching db.That structure maybe very useful for methods that take really long time to finish its stuff. The cons of the method is we have lots of separate log files and we have kind of high one to many transport communication,polling 100 minions for log information will not be very effcient.However it will work for single or a few machines very good !
