This shows you the differences between two versions of the page.

Link to this comparison view

hcmi:metrics-of-performance [2014/08/13 20:57] (current)
tlund1 created
Line 1: Line 1:
 +The two main metrics we are focusing on with respect to distributed organizations are robustness and responsiveness.
 +== Robustness ==
 +'''​Examples:'''​ Bio-inspired organizations and swarms are examples of robust organizations.
 +A Robust system should overcome these without much performance decrease.
 +* Outside Disturbances
 +* Internal Variation
 +* Agent Loss
 +* Faulty Information
 +Sensitivity to these abnormalities can be measured as the change in efficiency with respect to the performance criteria.
 +Additional measurements that should be low for robust systems.
 +* Time to Equilibrium
 +* Time to Local Consensus
 +* Overcommitment
 +* Wasted Resources
 +* Unproductive Effort
 +* Amount of State Thrashing
 +== Responsiveness ==
 +'''​Examples:'''​ Hierarchical organizations are usually very responsive.
 +Things a hierarchical organization should have.
 +* Centralized Goal, Objective, or Task
 +* Information Passing
 +* Tasks, Roles, and Resources assigned to Agents
 +Ways to increase efficiency in hierarchical organizations.
 +* Maximize Information Gain During Communication
 +* Reduce Coordination Costs 
 +* Reduce Social Loafing and Excess Idle Time
 +* Higher Level Decisions Affected by Information and Ideas from Lower Levels
 +== Futher Thoughts -- PC ==
 +From Sujit'​s Experiments.
 +* Responsiveness = Performance
 +* Robustness = Derivative of Performance w.r.t. Unreliability / The Unexpected ​
 +Robustness seems to need to be measured with respect to a type of unexpected phenomena. ​ A few examples of unexpected phenomena are listed above. ​ Since performance can be measure using a number of metrics, each performance metric can be tested for its robustness to a type of unexpected phenomena. ​ The graph of the robustness of a performance metric w.r.t. the degree of an unexpected phenomenon will result in an unreliability inpact curve analogous to a neglect impact curve. ​ From the board, we called this sensitivity. ​ But perhaps robustness is the sum total of such sensitivity curves.
 +The word responsiveness seems like it should measure how well the system responds to some change, and not just system performance. ​ Responsiveness could be measured as the time it takes to reach a type of organizational,​ informational,​ or environmental equilibrium after a task, role, goal, or organization change.
 +Perhaps performance should itself be a separate metric class.
 +From "​Developing Performance Metrics for the Supervisory Control of Multiple Robots"​ by Jacob W. Crandall and M. L. Cummings. (HRI2007)
 +* Metric classes suggested in paper:
 +** Interaction Efficiency
 +** Neglect Efficiency
 +** Attention Allocation Efficiency
 +* Necessities of metrics:
 +* Metrics should identify limits of all agents
 +* Metrics should have predictive power
 +* Metrics should contain key performance parameters that indicate overall effectiveness
 +From "​Identifying Generalizable Metic Classes to Evaluate Human-Robot Teams" by P. Pina, M. L. Cummings, J. W. Crandall, and M. Della Penna. (HRI2008)
 +* Metric Classes suggested in paper:
 +** Mission Effectiveness
 +** Human Behavior Efficiency
 +** Robot Behavior Efficiency
 +** Human Behavior Cognitive Precursors
 +** Human Behavior Physiological Precursors
 +** Collaborative Metrics
 +The metric classes in the earlier paper listed seem to be represented in the latter.
 +* Interaction Efficiency < Human Behavior Efficiency
 +* Neglect Efficiency < Robot Behavior Efficiency
 +* Attention Allocation Efficiency < Collaborative Metrics
 +It seems correct to split the behavior of the robot and human into different metrics. ​ Collaborative Metrics could also be split into human-human,​ human-robot,​ and robot-robot situations. ​ This could help identify (or predict) the location of a problem, whether it is a human problem ("​problem exists between keyboard and chair"​),​ an autonomy problem, or an organizational problem.
 +== Entropy -- PC ==
 +"​Entropy of activity, entropy of information,​ .. "
 +"The idea is that entropy should have a time-varying quality that satisfies some pattern."​
 +"​I'​m not sure what that pattern should be."
 +A few possible types of entropy for metrics:
 +* Behavioral Entropy
 +* Informational Entropy
 +* Organizational Entropy
 +* Task-Ability Entropy? ​ (Sort of a measure of how well the agents are suited for the types of tasks in the mission. ​ Based on affordances of agents, requirements of the tasks, and the environment)
 +What types and measurements of entropy would be useful in these situations?
 +Informational entropy over time could be a good way of measuring information flow and how information is distributed among agents.
 +Behavioral entropy could be a good way of measuring a human'​s state (from driving and measuring how sleepy people are).
 +What would organizational entropy be?  Would it require organizations to self-organize?​
hcmi/metrics-of-performance.txt ยท Last modified: 2014/08/13 20:57 by tlund1
Back to top
CC Attribution-Share Alike 4.0 International
chimeric.de = chi`s home Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0