Stemming Algorithm - AIT CSIM Program

Stemming Algorithm - AIT CSIM Program

Stemming Algorithms 9142608 9142609 From www.mis.nsysu.edu.tw/~syhwang/Courses/IR/StemmingAlgorithms.ppt, modified by Sumanta The Porter Algorithm Word = Stem + Affix(es)

E.g., generalizations = general + ization + s Stemming is the determination of the stem of a given word Porters stemmer is a rule-based algorithm

E.g., ational ate (apply: relational relate) Porters stemmer is heuristic, in that it is a practical method not guaranteed to be optimal The Porter Stemmer: Definitions Definitions CONSONANT: a letter other than A, E, I, O, U, and Y preceded by consonant (in TOY, consonants

are T,Y; in SYZYGY they are S, Z, G) VOWEL: any other letter With this definition all words and parts of words are of form : [C](VC)m[V] C=string of one or more consonants (con+) and [C] indicates arbitrary presence of the contents, i.e., possibly empty string as well. V=string of one or more vowels and [V] indicates arbitrary E.g., Troubles

C VC VC = C(VC)2 m is the measure of the word m = 0: TR, EE, TREE, Y, BY m = 1: TROUBLE, OATS, TREES, IVY m = 2: TROUBLES, PRIVATE, OATEN, ORRERY Spring 2002 NLE

3 Rule Format Rules are of the form (condition) S1 S2 where S1 and S2 are suffixes. Given a set of rules, only the one with the longest matching suffix S1 is applies. Conditions: 1. m --- measure of the stem m = k or m > k, where k is an integer

2.*X --- the stem ends with a given letter X 3.*v*--- the stem contains a vowel 4.*d --- the stem ends in double consonant 5.*o --- the stem ends with a consonant-vowel-consonant sequence, where the final consonant is not w, x or y, (e.g., wil, hop) Rules are divided into sets and in each successive step one set of rules is applied. Porter Steps Each step corresponds to a set of rules. The rules in

a step are examined in sequence , and only one rule from a step can apply { } step1a(word); step1b(stem); if (the second or third rule of step 1b was used)

step1b1(stem); step1c(stem); step2(stem); step3(stem); step4(stem); step5a(stem); step5b(stem); Should be: ti

if the second or third rule of step 1b was used Examples/Problems Step 1a Step 4 computers computer comput Step 1b singing sing

generalizations information instructor Try words of your own Porters Mishaps On-line Porters at http://textanalysisonline.com/nltk-porter-stemmer gives gas (noun) ga

gases (plural) gase gasses (verb, present tense) gass gassing (verb, present continuous) gass gaseous (adjective) gaseou This is not good all these words should ideally reduce to the same stem. Trade-off: More rules (accurate but slow) vs Less rules (efficient but sometimes wrong). Google does give different results for gas and gases, so maybe they use these Porter rules:-)

Recently Viewed Presentations

  • Title

    Title

    Is there irony in the title? Is this an imperative statement? Is "beloved" being used in the present or past tense? The Work's Acclaim. ... forming a multiethnic government to oversee the country's transition. 02/16/08. South Africa's Acclaim.
  • Identifying Quadratic Functions

    Identifying Quadratic Functions

    Fill the table of values to complete the table. Plot the points on a graph and draw the parabola. The equation is ?=2?2. Use the given x-values to complete. ... For quadratic functions, the parent function is ??=?2.
  • April 10th-14th

    April 10th-14th

    Assignment: read chapter 5 of LOTF online (google: read lord of the flies online) There will be a small quiz on the chapter on Wednesday to ensure that you have read it. Honors: we will read chapter 5 silently in...
  • Number Systems

    Number Systems

    7. Input / Output Chapt. 8
  • NEW Addition to our Retention and Success Initiative

    NEW Addition to our Retention and Success Initiative

    Students with (7) C-, D, F, W, NC, U grades at UCF will be placed in a probationary lack of progress status with the college. Students will meet with a CECS/COP advisor. Students with (10) C-,D, F, W, NC, U...
  • Engineering Drawing - PSAU

    Engineering Drawing - PSAU

    Times New Roman Arial Calibri Letter Gothic MT Default Design Engineering Drawing Line Conventions الشريحة 3 Lettering Sketching Pictorial الشريحة 7 الشريحة 8 Orthographic / Multiview الشريحة 10 Section Views Auxiliary Views الشريحة 13 الشريحة 14
  • Diapositiva 1 - WordPress.com

    Diapositiva 1 - WordPress.com

    AUTOR TÍTULO FICHAS Savater Fernando VALOR DE ELEGIR 5423 Cuauhtemoc Sanchez Carlos GRITO DESESPERADO 5422 Cuauhtemoc Sanchez Carlos DIRIGENTES DEL MUNDO FUTURO 5421 ...
  • Obesity in Adults Prevention and Management Recommendations 2015

    Obesity in Adults Prevention and Management Recommendations 2015

    Adult Obesity Prevention and Management Guidelines Objectives . Two separate guidelines were developed. These guidelines do not apply to those with a BMI >40 who may benefit from specialized services.