Wizer What-If Analyzer: Validation of Large Scale Stochastic

Wizer  What-If Analyzer: Validation of Large Scale Stochastic

Wizer What-If Analyzer: Validation of Large Scale Stochastic Agent Models

Why is Validation Hard?

Project Investigator:

Student:

Kathleen M. Carley CMU, ISRI, CASOS

Alex Yahja CMU, ISRI, CASOS

Response Surface and Wizer

Complexity of multi-agent systems: the significant

number of input parameters, output variables, & model
parameters, and their interactions
Models are a subset of reality: model assumptions may
not match assumptions underlying data
Cognitive bias: validation is often best not done by the
modeler(s)

Validation is knowledge intensive
Validation consumes a significant amount of time &
resources: large multi-agent systems take significant
time to run, even on a supercomputer making available
only a limited number of virtual experiments

The amount of empirical data is limited
Stochastic nature of many models adds to
complexity
Least developed area of the computational modeling

Response surface methodology: collection of mathematical
and statistical techniques for the modeling & analysis of
problems in which a response of interest is influenced by
several variables.
Wizer:
extends response surface methodology by performing
knowledge-intensive search steps via a social inference
engine, utilizing laws of logic, instead of just doing
mathematical & statistical calculations
integration of inference engine & simulation virtual
experiments
Wizers testbed: Wizer is used to validate BioWar a
scalable multi-agent network simulator for disease spread
in a heterogeneous population. Biowar -- which has a
complex response surface -- can be viewed as a multidimensional numeric & symbolic optimization problem: e.g.,
school absenteeism is influenced by student health status
and weather announcements.

Conceptual View of Wizer
How Validation is Currently
Done

In software engineering:
Automated program verification and testing
Theorem proving: mechanization of formal

Work
absence

Real
Statistics
Alert Wizer

Doctor
visits

Calculate
Statistics

ER visits

Minimum Bound
Check

Compare
Statistics
For Simulated
with Real

Maximum Bound
Check

Simulation outputs

ER visits
(SDI)
7 Drug Types
Purchases

BioWar
simulator

Mean Significantly
Not Different
Check

EMPIRICAL
DATA

Soft knowledge

reasoning, following laws of logic
Model checking: method for formally verifying
finite-state concurrent systems using temporal
logic.

Wizer is a data-driven automation of
validation, utilizing the integration of
inference engine and simulation to perform
knowledge-intensive what-if analyses
through simulations and inferences. Hence
the name Wizer: What-if Analyzer.

AUTOMATED
CHECK
REPORT

School
absence

Simulation nuggets
Experiment Designer
Simulation model &
parameters for each
knowledge nugget
constrained by soft
knowledge
New experiment
specification

exploratory modeling

Wizer

original thresholds governing agent behaviors:
th0=5, th1=20, th2=130, th3=260
(these thresholds interact to determine whether an agent
goes certain places)
Checking mean & std. dev. of visits to places:
Work:
mean 24880.8, std. dev. 15741.7
School:
mean 6499.44, std. dev. 4115.47
Pharmacy: mean 1427.91, std. dev. 762.439
Doctor:
mean 199.369, std. dev. 146.913
Emergency_Room: mean 30.1935, std. dev. 24.1886
work is outside bound
threshold th0 is the actual cause of work being too low
work is too low, decrease th0
school is within bound
pharmacy is within bound
doctor is within bound
Emergency_Room is outside bound
thresholds th2 and th3 are the actual causes of ER
being too high
Emergency_Room is too high, increase th2, increase th3
Wizer modified thresholds:
th0=3, th1=20, th2=132, th3=262

System Diagram

Manually
Human expert judgments (pitfall: implicit biases)
Assisted by computer: Monte-Carlo technique and

In engineering:
Experiment design
Response surface methodology

Wizer 0.1 Result with BioWar

Inner Workings of Alert Wizer

Meta-Modeler

Flow Diagram of Wizer
Alert
Wizer

Which
data streams
are wrong/right
and how

New parameters

Knowledge as
simulation

Knowledge nuggets

System information
of what parameters
influence which data
stream & how

Software engineering
knowledge

New codes
Simulation
histories *

Trend Inference
Engine
Soft knowledge

Causal Inference
Engine

Trends and
differentials
Response Surface
Comparator

Simulation outputs

Causal relations
Causal Detector

Simulation
happenings *

Soft knowledge

Control commands
Feedback

Simulation History
Organizer
Empirical constraints
on parameters

Knowledge
nuggets and soft
knowledge

Knowledge nuggets

New execution
Automated code commands Experiment Executor
generator

Wizer
Inference Engine,
inferring and deciding
which parameters to change
and how

Knowledge miner &
causal relation
extractor

Patterns, norms,
Knowledge base
constraints, culture, and
other soft knowledge Soft knowledge
inference engine

New multiagent model

Empirical data from
literature, journals, surveys,
census, health care,
sociology, epidemiology,
geography, software
engineering, etc.

Simulator
Simulation
happenings *

* Performance, old multi-agent architecture, old experiment specification, and results

Recently Viewed Presentations

  • PHYSICIST PROFILE Chief Executive Officer American Physical Society

    PHYSICIST PROFILE Chief Executive Officer American Physical Society

    Blackbody Source. Hot Gas (e.g. nebula) Star (blackbody + gas atmosphere) Emission Spectrum. Absorption Spectrum. Scientists learn about distant objects (such as stars or interstellar gas) by looking at the object's light, broken down into its component wavelengths (similar to...
  • El abecedario

    El abecedario

    El alfabeto español A ah B be grande C ce CH che D de E eh F efe G heh H ah-che I eeee J hota K ka L ele LL eh-ye M eme N ene Ñ eh-nye O oh...
  • Hurricane Retrieval Algorithm for the SeaWinds Scatterometer

    Hurricane Retrieval Algorithm for the SeaWinds Scatterometer

    NASA Ocean Vector Winds Science Team sponsored CFRSL to develop improved QuikSCAT wind retrieval algorithm for extreme wind events. This PI OVW product is known as . Q-Winds. Combines active/passive measurements from SeaWinds. Tuned to HRD's H*Wind surface wind analysis...
  • Introduction to Astronomy

    Introduction to Astronomy

    This heat allows fusion to occur in a shell of material surrounding the core… Due to the higher central temperature, the star's luminosity is greater than before… This increased energy production causes the outer part of the star to expand...
  • Unit 1: From West Africa to the Early Americas (Ancient Times ...

    Unit 1: From West Africa to the Early Americas (Ancient Times ...

    Unit 1: From West Africa to the Early Americas (Ancient Times - 1763) ... Eventually Ghana's rulers converted to Islam and learned Arabic, much of the population never converted. In 1076 Muslim Almoravids conquered Ghana and disrupted the gold-salt trade.
  • Basic Theatre Terminology - Mesa Public Schools

    Basic Theatre Terminology - Mesa Public Schools

    Basic Theatre Terminology. Beginning Drama. Stage Fright. A nervous feeling felt by someone who is going to appear in front of an audience. Drama. A piece of writing that tells a story and is performed on stage. Energy.
  • Prefixes and Suffixes

    Prefixes and Suffixes

    What are they and why do we need to know them? Prefix - beginning Root - main word Suffix - ending What is a root word? word you can add beginnings and ends to Why look at root word? helps...
  • Forming Verb Tenses for Regular Verbs

    Forming Verb Tenses for Regular Verbs

    Forming Verb Tenses for Regular Verbs Mrs. Zakrewski 7th Grade Language Arts To form the PRESENT TENSE…use the base form of the verb. Examples: complain, worry, enjoy, save PRESENT TENSE VERBS are verbs that are existing or happening now.