Principal Investigator C Dewey
Text documents generally contain two forms of structures, logical structures and physical structures. Loosely speaking, logical structures are sections of text that are both visually and semantically distinct. For example, a document may have an "introduction", a "body", and a "conclusion" as its logical structures. These structures are so named because each section has a distinct purpose in conveying the document's logical argu-ments or intentions. Perfect machine recognition of logical structures in large collections of documents is an unsolved problem in computational linguistics.
This thesis presents evidence that a new family of functions on text segments carries information that is useful for differentiating document logical structures. For any given text segment, a function of this form is referred to as the cadence, and it is based on a new interpretation of the vector space representation that Gerard Salton introduced in 1975. To test the hypothesis that the cadence of a text segment carries information helpful to differentiating logical structures, a corpus was built containing 2800 documents with manually-annotated logical structures. Structures representing abstracts, introductions, bodies, and conclusions from this corpus were clustered with a k-means algorithm using cadence data. Precision and recall performances were computed for the results, and a chi-squared goodness-of-fit test was used to determine the statistical significance of the clustering results.