PHP Sentence Boundaries Detection

PHP sentence boundaries including empty lines?

The end of the pattern should be replaced with:

  (?:\h+|^$)          # Split on whitespace between sentences\/empty lines.
/mix';

See IDEONE demo

Note that [\s+|^$] really matches whitespace (both horizontal and vertical, like newlines), +, |, ^ and $ symbols because it is a character class.

Instead of a character class, a group (better, non-capturing here) is necessary. Inside a group (marked with (...)) the | works as an alternation operator.

Instead of \s, I suggest using \h that matches horizontal whitespace (no linebreaks) only.

The ^$ will only match an empty string if no /m multiline modifier is used. So, I have added /m modifier to the options.

And note that I had to escape the / inside the last comment, otherwise there was a warning that the regex is incorrect. Or, use different regex delimiters.

Regex sentence content matching

I am the author of the cited sentence splitting answer. Here's a modified version that may suit your purposes:

An enhanced regex solution

Assuming you do care about handling: Mr. and Mrs. etc. abbreviations, then the following single regex solution works pretty well:

<?php // test.php Rev:20140218_1500
$re = '/# Match sentence ending in .!? followed by optional quote.
    (                  # $1: Sentence.
      [^.!?]+          # One or more non-end-of-sentence chars.
      (?:              # Zero or more not-end-of-sentence dots.
        \.             # Allow dot mid-sentence, but only if:
        (?:            # Group allowable dot alternatives.
          (?=[^\s\'"]) # Dot is ok if followed by non-ws,
        | (?<=         # or not one of the following:
            Mr\.       # Either "Mr."
          | Mrs\.      # or "Mrs.",
          | Ms\.       # or "Ms.",
          | Jr\.       # or "Jr.",
          | Dr\.       # or "Dr.",
          | Prof\.     # or "Prof.",
          | Sr\.       # or "Sr.",
          | T\.V\.A\.  # or "T.V.A.",
                       # or... (you get the idea).
          )            # End positive lookbehind.
        )              # Group allowable dot alternatives.
        [^.!?]*        # Zero or more non-end-of-sentence chars.
      )*               # Zero or more not-end-of-sentence dots.
      (?:              # Sentence end alternatives.
        [.!?]          # Either end of sentence punctuation
        [\'"]?         # followed by optional quote,
      | $              # Or end of string with no punctuation.
      )                # Sentence end alternatives.
    )                  # End $1: Sentence.
    (?:\s+|$)          # Sentence ends with ws or EOS.
    /ix';

$text = 'This is sentence one. Sentence two! Sentence thr'.
        'ee? Sentence "four". Sentence "five"! Sentence "'.
        'six"? Sentence "seven." Sentence \'eight!\' Dr. '.
        'Jones said: "Mrs. Smith you have a lovely daught'.
        'er!" The T.V.A. is a big project! Last sentence '.
        'with no ending punctuation';

$sentences = array(); // Initialize array of sentences.

function _getSentencesCallback($matches) {
    global $sentences;
    $sentences[] = $matches[1];
    return '';
}
preg_replace_callback($re, '_getSentencesCallback', $text);
for ($i = 0; $i < count($sentences); ++$i) {
    printf("Sentence[%d] = [%s]\n", $i + 1, $sentences[$i]);
}
?>

Note that you can easily add or take away abbreviations from the expression. Given the following test paragraph:

This is sentence one. Sentence two! Sentence three? Sentence "four". Sentence "five"! Sentence "six"? Sentence "seven." Sentence 'eight!' Dr. Jones said: "Mrs. Smith you have a lovely daughter!" The T.V.A. is a big project!

Here is the output from the script:

Sentence[1] = [This is sentence one.]
Sentence[2] = [Sentence two!]
Sentence[3] = [Sentence three?]
Sentence[4] = [Sentence "four".]
Sentence[5] = [Sentence "five"!]
Sentence[6] = [Sentence "six"?]
Sentence[7] = [Sentence "seven."]
Sentence[8] = [Sentence 'eight!']
Sentence[9] = [Dr. Jones said: "Mrs. Smith you have a lovely daughter!"]
Sentence[10] = [The T.V.A. is a big project!]
Sentence[11] = [Last sentence with no ending punctuation]

Hope this helps and Happy Regexing!

Edit: 2014-02-19 08:00 Last sentence at end of string no longer requires punctuation.

Split string into sentences using regex

As it should be expected, any sort of natural language processing is not a trivial task. The reason for it is that they are evolutionary systems. There is no single person who sat down and thought about which are good ideas and which - not. Every rule has 20-40% exceptions. With that said the complexity of a single regex that can do your bidding would be off the charts. Still, the following solution relies mainly on regexes.

---

• The idea is to gradually go over the text.
• At any given time, the current chunk of the text will be contained in two different parts. One, which is the candidate for a substring before a sentence boundary and another - after.
• The first 10 regex pairs detect positions which look like sentence boundaries, but actually aren't. In that case, before and after are advanced without registering a new sentence.
• If none of these pairs matches, matching will be attempted with the last 3 pairs, possibly detecting a boundary.

---

As for where did these regexes come from? - I translated this Ruby library, which is generated based on this paper. If you truly want to understand them, there is no alternative but to read the paper.

As far as accuracy goes - I encourage you to test it with different texts. After some experimentation, I was very pleasantly surprised.

In terms of performance - the regexes should be highly performant as all of them have either a \A or \Z anchor, there are almost no repetition quantifiers, and in the places there are - there can't be any backtracking. Still, regexes are regexes. You will have to do some benchmarking if you plan to use this is tight loops on huge chunks of text.

---

Mandatory disclaimer: excuse my rusty php skills. The following code might not be the most idiomatic php ever, it should still be clear enough to get the point across.

---

function sentence_split($text) {
    $before_regexes = array('/(?:(?:[\'\"„][\.!?…][\'\"”]\s)|(?:[^\.]\s[A-Z]\.\s)|(?:\b(?:St|Gen|Hon|Prof|Dr|Mr|Ms|Mrs|[JS]r|Col|Maj|Brig|Sgt|Capt|Cmnd|Sen|Rev|Rep|Revd)\.\s)|(?:\b(?:St|Gen|Hon|Prof|Dr|Mr|Ms|Mrs|[JS]r|Col|Maj|Brig|Sgt|Capt|Cmnd|Sen|Rev|Rep|Revd)\.\s[A-Z]\.\s)|(?:\bApr\.\s)|(?:\bAug\.\s)|(?:\bBros\.\s)|(?:\bCo\.\s)|(?:\bCorp\.\s)|(?:\bDec\.\s)|(?:\bDist\.\s)|(?:\bFeb\.\s)|(?:\bInc\.\s)|(?:\bJan\.\s)|(?:\bJul\.\s)|(?:\bJun\.\s)|(?:\bMar\.\s)|(?:\bNov\.\s)|(?:\bOct\.\s)|(?:\bPh\.?D\.\s)|(?:\bSept?\.\s)|(?:\b\p{Lu}\.\p{Lu}\.\s)|(?:\b\p{Lu}\.\s\p{Lu}\.\s)|(?:\bcf\.\s)|(?:\be\.g\.\s)|(?:\besp\.\s)|(?:\bet\b\s\bal\.\s)|(?:\bvs\.\s)|(?:\p{Ps}[!?]+\p{Pe} ))\Z/su',
        '/(?:(?:[\.\s]\p{L}{1,2}\.\s))\Z/su',
        '/(?:(?:[\[\(]*\.\.\.[\]\)]* ))\Z/su',
        '/(?:(?:\b(?:pp|[Vv]iz|i\.?\s*e|[Vvol]|[Rr]col|maj|Lt|[Ff]ig|[Ff]igs|[Vv]iz|[Vv]ols|[Aa]pprox|[Ii]ncl|Pres|[Dd]ept|min|max|[Gg]ovt|lb|ft|c\.?\s*f|vs)\.\s))\Z/su',
        '/(?:(?:\b[Ee]tc\.\s))\Z/su',
        '/(?:(?:[\.!?…]+\p{Pe} )|(?:[\[\(]*…[\]\)]* ))\Z/su',
        '/(?:(?:\b\p{L}\.))\Z/su',
        '/(?:(?:\b\p{L}\.\s))\Z/su',
        '/(?:(?:\b[Ff]igs?\.\s)|(?:\b[nN]o\.\s))\Z/su',
        '/(?:(?:[\"”\']\s*))\Z/su',
        '/(?:(?:[\.!?…][\x{00BB}\x{2019}\x{201D}\x{203A}\"\'\p{Pe}\x{0002}]*\s)|(?:\r?\n))\Z/su',
        '/(?:(?:[\.!?…][\'\"\x{00BB}\x{2019}\x{201D}\x{203A}\p{Pe}\x{0002}]*))\Z/su',
        '/(?:(?:\s\p{L}[\.!?…]\s))\Z/su');
    $after_regexes = array('/\A(?:)/su',
        '/\A(?:[\p{N}\p{Ll}])/su',
        '/\A(?:[^\p{Lu}])/su',
        '/\A(?:[^\p{Lu}]|I)/su',
        '/\A(?:[^p{Lu}])/su',
        '/\A(?:\p{Ll})/su',
        '/\A(?:\p{L}\.)/su',
        '/\A(?:\p{L}\.\s)/su',
        '/\A(?:\p{N})/su',
        '/\A(?:\s*\p{Ll})/su',
        '/\A(?:)/su',
        '/\A(?:\p{Lu}[^\p{Lu}])/su',
        '/\A(?:\p{Lu}\p{Ll})/su');
    $is_sentence_boundary = array(false, false, false, false, false, false, false, false, false, false, true, true, true);
    $count = 13;

    $sentences = array();
    $sentence = '';
    $before = '';
    $after = substr($text, 0, 10);
    $text = substr($text, 10);

    while($text != '') {
        for($i = 0; $i < $count; $i++) {
            if(preg_match($before_regexes[$i], $before) && preg_match($after_regexes[$i], $after)) {
                if($is_sentence_boundary[$i]) {
                    array_push($sentences, $sentence);
                    $sentence = '';
                }
                break;
            }
        }

        $first_from_text = $text[0];
        $text = substr($text, 1);
        $first_from_after = $after[0];
        $after = substr($after, 1);
        $before .= $first_from_after;
        $sentence .= $first_from_after;
        $after .= $first_from_text;
    }

    if($sentence != '' && $after != '') {
        array_push($sentences, $sentence.$after);
    }

    return $sentences;
}

$text = "Mr. Entertainment media properties. Fairy Tail 3.5 and Tokyo Ghoul.";
print_r(sentence_split($text));

Sentence boundary detection in HTML

The solution I implemented was 1. split the document into separate blocks on all html tags except the inline tags (<i>, <b>, <span>, etc.), 2. strip the inline tags from each block, 3. look for sentences within each block using traditional methods.

Using Regular Expression to create arrays of sentences from text content

The best answer I have found for creating arrays of coherent sentences is the regex solution found in the link suggested by @Marc in the comments above.

The best thing about this regex is that you can add to it. For example I've added abbreviations for months such as SEPT. which are typically followed with fullstops.

https://stackoverflow.com/a/7438782/3662086

Explode a paragraph into sentences in PHP

You can do:

preg_split('/\.|\?|!/',$mystring);

or (simpler):

preg_split('/[.?!]/',$mystring);

---

Related Topics