SMBL and Blast - University of Wisconsin-Madison

SMBL and Blast - University of Wisconsin-Madison

SMBL and Blast Joe Rinkovsky Unix Systems Support Group Indiana University Introduction

IU has around 2000 Windows PCs in public Student Technology Centers Condor is used to harvest unused cycles Simple Message Brokering Library(SMBL) used for parallelizing applications on Windows Web portal for user interaction Project History

[email protected] Was used as initial test of Condor SMBL was created to address the lack of a general purpose parallel library on Windows that could tolerate sporadically available systems FastDNAml was ported to SMBL Web portal created Other apps ported to SMBL(MEME,BLAST) System Architecture

Condor server running on Linux BLAST databases served via Samba on a second Linux machine Apache/MySQL/PHP web portal Windows clients What is SMBL?

Simple Message Brokering Library Open Source( Uses master / worker model Process and Port Manager(PPM) manages SMBL servers and master processes Number of master /foreman processes is different for each application

SMBL workers contact the SMBL master to get work SMBL server terminates workers when they are no longer needed Condor and SMBL

Condor is used as the scheduling and delivery system for SMBL workers SMBL workers contact the SMBL server when they start running to begin receiving work. SMBL server seperates the work to be into smaller pieces depending on the number of workers Work is redistributed if a worker is lost SMBL server terminates workers when there is no work left Applications using SMBL

FastDNAml Generates phylogenic trees from molecular data MEME Detects patterns in nucleotide and protein sequences NCBI BLAST(blastall) Query molecular sequences against sequence databases The Challenges of porting BLAST to SMBL

BLAST relies on the availability of large database files Files too large for efficient delivery via Condor Local copies of databases on pool machines would be difficult to manage Sharing DB files via Samba is the best solution Samba was moved to a seperate server to increase perfomance

The Challenges of porting BLAST to SMBL(cont.) BLAST jobs take more time to complete than FastDNAml and MEME Dissapearing

worker problem Pool machines would end up in CLAIMED/IDLE state Size of our Condor pool made the problem hard to track Only jobs taking more than 30 minutes were affected Problem was determined to be state table sessions timing out on the machine room firewall. Machines were removed from firewall and switched to host-based iptables firewall.

Web portal Apache/MySQL/PHP based Jobs are submitted via portal ONLY Condor submit files are dynamically generated based on user input

Status of jobs can be checked using the portal Results retrieved from the portal Questions?

Recently Viewed Presentations

  • Lecture # 6: Carbohydrates and Lipids (Chapter 2)

    Lecture # 6: Carbohydrates and Lipids (Chapter 2)

    2- Define hydrophilic/polar and hydrophobic/nonpolar. ... 4- Eicosanoids. 5- Steroids. A fatty acid is a chain of 4 to 24 carbon atoms with a carboxyl (acid) group on one end, methyl group on the other and hydrogen bonded along the...
  • Multipath TCP - WINLAB

    Multipath TCP - WINLAB

    Multipath TCP Windows. Multipath TCP maintains one window per Multipath TCP connection. Windowis relative to the last acked data (Data Ack) Windowissharedamong all subflows. It's up to the implementation to decide how the windowisshared. Windowistransmittedinside the windowfield of the regular...
  • SIUC New Employee Orientation

    SIUC New Employee Orientation

    There will be no changes to your health plan options this Benefit Choice Period. If you wish to keep your coverage, no action is needed unless you intend to enroll or re-enroll in a Flexible Spending Account. If you wish...
  • Agenda Welcome and Introductions Thank You(s) Awards &

    Agenda Welcome and Introductions Thank You(s) Awards &

    Jill Mion (Barber/Cos) Reggie Mosher (IT) Dong Ngo (Math of Finance) Salwa Rashad (Math of Finance) Dennis Thompson (Finance/Accounting) Rick Trummer (Photography) Thank you . Tom Thompson - Accounting. Peter Wilmarth - Marketing and Recreation Management.
  • What is Geography About?

    What is Geography About?

    Think about these key words: Task… Use your keywords to write a definition (an explanation) of what the word geography means. You may wish to start your sentence in one of these ways: 'Geography means….' 'The word geography describes….' 'A...
  • Les reconnaissez-vous

    Les reconnaissez-vous

    Images du Net Musique : Diégo Modena - Song of Ocarina Réalisé par AKA3 (Limité aux francophones par AG) Alain Chamfort Chanteur français Alain Delon Acteur français Alain Souchon Chanteur français Arielle Dombasle Actrice - chanteuse française Avril Lavigne Chanteuse...
  • Voting System Technologies

    Voting System Technologies

    Longitude Prize 2014: Crowdsourced Inspiration Ensure everyone can have access to safe & clean water? Restore movement to those with paralysis? Fly without damaging the environment? Help people with dementia live independently for longer Prevent the rise of resistance to...
  • Sonnet: Composed Upon Westminster Bridge

    Sonnet: Composed Upon Westminster Bridge

    Author Biography. Composed upon Westminster Bridge, September 3, 1802 is a sonnet by William Wordsworth describing London and the River Thames, viewed from Westminster Bridge in the early morning.. Wordsworth and his sister Dorothy were travelling to Calais to visit...