NIST Big Data Working Group

NIST Big Data Working Group

NIST Big Data Public Working Group Reference Architecture Subgroup September 30, 2013 Co-chairs: Orit Levin Microsoft James Ketner AT&T Don Krapohl Augmented Intel Agenda Deliverable #1: White Paper: Survey of Existing Big Data RAs Deliverable #2: NIST Big Data Reference Architecture Next Steps Reference Architecture Subgroup 2 NIST White Paper Survey of Big Data Architecture Models Input Document M0151 Reference Architecture Subgroup 3 List Of Surveyed Architectures Vendor-neutral and technology-agnostic proposals

Bob Marcus ET-Strategies Orit Levin Microsoft Gary Mazzaferro AlloyCloud Yuri Demchenko University of Amsterdam Vendors Architectures IBM Oracle Booz Allen Hamilton EMC SAP 9sight LexusNexis Reference Architecture Subgroup 4 Vendor-neutral and Technology-agnostic Proposals Data Processing Flow M0039

Data Transformation Flow M0017 Reference Architecture Subgroup IT Stack M0047 5 Vendor-neutral and Technology-agnostic Proposals Data Processing Flow M0039 Data Transformation Flow M0017 Reference Architecture Subgroup IT Stack M0047 6 Vendor-neutral and Technology-agnostic Proposals Data Processing Flow M0039

Data Transformation Flow M0017 Reference Architecture Subgroup IT Stack M0047 7 Vendor-neutral and Technology-agnostic Proposals Data Processing Flow M0039 Data Transformation Flow M0017 Reference Architecture Subgroup IT Stack M0047 8 Draft Agreement / Rough Consensus Transformation Usage

Reference Architecture Subgroup Network Data stores In-memory DBs Analytic DBs Cloud Computing Data Infrastructure includes Management Sources Security Processing functions Analytic functions Visualization functions Data Infrastructure Transformation includes 9 NIST BIG DATA Reference Architecture Input Document M0226 Reference Architecture Subgroup

10 What the Baseline Big Data RA Is Is Not A superset of a traditional data system A representation of a vendor-neutral and technology-agnostic system A functional architecture comprised of logical roles Applicable to a variety of business models Tightly-integrated enterprise systems Loosely-coupled vertical industries A business architecture representing internal vs. external functional boundaries A deployment architecture A detailed IT RA of a specific system implementation All of the above will be developed in the next stage in the context of specific use cases.

Reference Architecture Subgroup 11 System Orchestrator Data Provider Big Data Application Provider Analytic processing of data Transfer of data Code execution on data et situ Storage, retrieval, search, etc. of data Providing computing infrastructure Providing networking infrastructure Etc. Data Consumer Main Big Data Functional

Frameworks Blocks Big Data Framework Processing Frameworks (analytic tools, etc.) Provider Horizontally Scalable Vertically Scalable Platforms (databases, etc.) Horizontally Scalable Vertically Scalable Infrastructures Horizontally Scalable (VM clusters) Vertically Scalable Physical and Virtual Resources (networking, computing, etc.) Reference Architecture Subgroup 12 Main Big Data Functional Application Blocks Provider System Orchestrator Collection Collection Curation Curation

Visualizat Visualizat ion Analytics ion Analytics Access Access Data Consumer Data Provider Big Data Application Provider Big Data Framework Processing Frameworks (analytic tools, etc.) Provider Horizontally Scalable Vertically Scalable Platforms (databases, etc.) Horizontally Scalable Vertically Scalable Infrastructures Horizontally Scalable (VM clusters) Vertically Scalable Physical and Virtual Resources (networking, computing, etc.)

Reference Architecture Subgroup 13 Application Specific Identity Management & Authorization Etc. MainData Big Functional Flow Frameworks Blocks System Orchestrator Collection Collection Curation Curation Visualizat Visualizat ion Analytics ion Analytics

Access Access DA TAS W S W W Big Data Framework Processing Frameworks (analytic tools, etc.) Provider Discovery of data Description of data Horizontally Scalable Access to data Code execution on data Platforms (databases, etc.) Etc. Data Consumer DA TAS DAT DAT A A

Data Data Provider Provider Big Data Application Provider Vertically Scalable Horizontally Scalable Discovery of services Description of data Visualization of data Rendering of data Reporting of data Code execution on data Etc. Vertically Scalable Infrastructures Horizontally Scalable (VM clusters) Vertically Scalable Physical and Virtual Resources (networking, computing, etc.)

Reference Architecture Subgroup 14 Security & Privacy (& Management) System Orchestrator Access Access DA TAS W S W W Big Data Framework Processing Frameworks (analytic tools, etc.) Provider Horizontally Scalable Vertically Scalable Platforms (databases, etc.) Horizontally Scalable Vertically Scalable Infrastructures Horizontally Scalable (VM clusters) Vertically Scalable Security &

P rai n va ag cy M ement Collection Collection Curation Curation Visualizat Visualizat ion Analytics ion Analytics Data Consumer DA TAS DAT DAT A A Data Provider Big Data Application Provider

Physical and Virtual Resources (networking, computing, etc.) Reference Architecture Subgroup 15 I N F O R M AT I O N VA L U E CHAIN System Orchestrator Access Access W Big Data Framework Processing Frameworks (analytic tools, etc.) Provider Horizontally Scalable Vertically Scalable Service Use DA TA Data S Flow Analytic W Tools

Transfe r Platforms (databases, etc.) Horizontally Scalable Vertically Scalable Infrastructures Horizontally Scalable (VM clusters) Vertically Scalable I T VA L U E S W W KEY: DA TAS Security & P rai n va ag cy M ement Collection Collection Curation Curation

Visualizat Visualizat ion Analytics ion Analytics Data Consumer DA TAS DAT DAT A A Data Provider Big Data Application Provider Physical and Virtual Resources (networking, computing, etc.) Reference Architecture Subgroup 16 Big Data Reference Architecture V1.0 Outline Executive Summary 1 Introduction 2 Big Data System Requirements

3 Conceptual Model 4 Main Components 4.1 Data Provider 4.2 Big Data Application Provider 4.3 Big Data Framework Provider 4.4 Data Consumer 4.5 System Orchestrator 5 Management 5.1 System Management 5.2 Lifecycle Management 6 Security and Privacy 7 Big Data Taxonomy Appendix A: Terms and Definitions Appendix B: Acronyms Appendix C: References Appendix D: Deployment Considerations 1 Big Data Framework Provider 1.1 Traditional On-Premise Frameworks 1.2 Cloud Service Providers Reference Architecture Subgroup 17 Summary Summary The draft of the NIST White Paper: Survey of Existing Big Data RAs (v.1.2) is available as M0151v3 The draft of the NIST Big Data functional reference architecture (RA v.1.0) is available as M0226v8 Next Steps

Continue the editorial and alignment effort Map generic Big Data use cases to RA Map specific collected Big Data cases to RA Lets exchange additional ideas this afternoon at the breakout session! Reference Architecture Subgroup 18 THANK YOU Co-chairs: Orit Levin Microsoft James Ketner AT&T Don Krapohl Augmented Intel Reference Architecture Subgroup 19

Recently Viewed Presentations

  • Area of triangle

    Area of triangle

    Find the base if its height is 9m. Solution: Given: Area(A) = 63m2 , Height(h) = 9m To find: Base(b) Area of a parallelogram = b x h sq units 63m2 = b x 9 b x 9 = 63m2...
  • www.ucd.ie

    www.ucd.ie

    Rugby, Women's Rugby, Soccer, Women's Soccer. Mountaineering, Orienteering, Caving & Potholing, Snowsports. ... levies and Mazars admin. charge will be passed to the respective club for each coach employed by them. ... Please take note of this link:
  • Aside on AI Will computers ever be intelligent?

    Aside on AI Will computers ever be intelligent?

    Statistical machine learning. Gather much training data. Design a statistical model. Train, train, train. Given new instances, use model to categorize or predict
  • Studying Rigorously Defined Health Care Processes Describe, capture,

    Studying Rigorously Defined Health Care Processes Describe, capture,

    Formally Model and Analyze Processes Using Little-JIL Capture How Real Individuals Complete Complex Processes Define a process - such as verifying a patient's identity - at any level of detail Specify how to handle exceptional, non-normative conditions Use the model...
  • Tahap-tahap Pembentukan Tim Kerjasama

    Tahap-tahap Pembentukan Tim Kerjasama

    Tim ini mandiri dalam arti memiliki kewenangan mengatur dirinya sendiri, serta menentukan sendiri apa yang perlu mereka lakukan. Mereka juga memiliki akses langsung pada informasi yang memungkinkan mereka untuk merencanakan, mengendalikan dan memperbaiki kerja mereka. ...
  • TH19 - strtn.org

    TH19 - strtn.org

    * L'évaluation a porté sur : 1) le caractère conclusif de l'examen 2) la présence ou non d'une embolie pulmonaire et sa topographie 3) les signes de gravité en cas d'EPA: rapport VD/VG 4) La présence d'une pathologie associée ou...
  • African Religions YORUBA CULTURE WHO ARE YORUBA PEOPLE?

    African Religions YORUBA CULTURE WHO ARE YORUBA PEOPLE?

    Ile-Ife The Cradle of Yoruba Civilization African Religions "African Traditional Religions" = indigenous religions Islam = introduced to sub-Saharan AF in 11th c. Christianity = introduced to West AF in 15th c. Syncretic = indigenous Christian movements since early 1900's...
  • KCCT Review - Warren County Public Schools

    KCCT Review - Warren County Public Schools

    Context Clues *When you don't know the meaning of a word, use CONTEXT CLUES to help figure it out. The context is the words, sentences, and paragraphs surrounding an unknown word. YOU TRY! 1. Joan loves to buy exotic foods:...