Tisane is a natural language processing service, providing:

• standard NLP functionality
• special functions for detection of problematic or abusive content
• functionality for law enforcement / eDiscovery / intelligence applications
• low-level NLP like morphological analysis and tokenization of no-space languages (Chinese, Japanese, Thai)

Tisane was engineered from the ground up to parse user-generated content and tackle "dirty NLP".

Under the hood, Tisane is a multifunctional NLU system, like spaCy, but based on a monolithic architecture, and with a focus on user-generated content. All the functions are exposed using the same language models and the same analysis process invoked using the POST /parse method. In just one call you're getting entities, sentiment, topic, problematic content, and low-level NLP data. Other vendors force you to issue a separate call for each of these functions. If you need to detect sentiment, entities, and topics, with Tisane you end up making 3 times less calls.

Tisane is available as SaaS or an on-prem solution.

Entity Extraction

These are the types of the entities Tisane extracts:

Detected entities are located under the entities_summary section.

Language Identification

Detects the language of the utterance. It is possible (with reservation) to segment a fragment using several languages.

To invoke automatic detection when parsing or translating, use * as a language code. Use lang_detect_segmentation_regex setting to define custom language detection fragment boundaries. For example, if multiple languages may be used in different sentences in the same text, define the regex as: (([\r\n]|[.!?][ ])) .

To detect the language only, invoke POST detectLanguage method.

Sentiment Analysis

Tisane provides sentiment analysis both at the document level and as a breakdown by aspects / facets (so-called "aspect-based sentiment analysis"). If you're looking to understand what exactly the customers like or don't like about your product offering, the breakdown is stored in the sentiment_expressions section.

If you're looking for a document-level sentiment score, set the document_sentiment toggle to true in settings. Also, if you're processing reviews, set format to review for better results.

Note that sentiment analysis and detection of problematic content are not the same. Sentiment can be negative and not be a personal attack or hate speech; on the other hand, criminal activity or sexual advances may not necessarily carry negative sentiment.

Topics

Tisane extracts topics (subjects) in IAB, IPTC standards, as Wikidata IDs, or native Tisane labels / family IDs.

If you prefer more topics, set optimize_topics toggle to true.

Problematic Content

Tisane tags the following types of problematic content:

• personal_attack - cyberbullying, ad hominem attacks, insults (not necessarily involving profanities)
• bigotry - hate speech targeting protected classes (not political factions)
• sexual_advances
• profanity - plain or obfuscated profanities
• criminal_activity - attempts to buy or sell illegal items, stolen data, documents, etc. (Not conversations about crime)
• allegation - claims about objectionable conduct
• external_contact - attempts to establish contact (e.g. invitations to exchange phones, emails, handles or set up meetings)
• mental_issue - mental issues and suicidal ideation
• disturbing - disturbing depictions of violence
• adult_only - adult only topics (e.g. in minors-only communities)
• provocation - attempts to provoke a community

...and more, supporting slang and misspellings.

Low-level Data

With the low-level data, you can:

• segment text in languages without spaces (Chinese, Japanese, Thai)
• decompound text in languages using compounds or agglutination (German, Dutch, Norwegian, Hungarian, Turkish)
• detect multiword expressions (MWEs) and lexical chunks
• find noun phrases, verb phrases, prepositional phrases, main clauses, etc.
• tag parts of speech
• get extensive grammatical info, e.g. verb tense, plurality, gender
• detect questions, negations
• find semantic roles (e.g. agents and patients) in active and passive voice
• split sentences
• locate categories of concepts not included in entity types

To obtain parse trees and individual words, set parses or words toggles to true.

Translation

Built-in machine translation (POST /transform) is intended to provide unfiltered translation for potentially offensive, profane, and slang-abundant content.

Supported Languages

Currently we support over 30 languages:

• English
• Afrikaans
• Albanian
• Arabic
• Danish
• German
• Spanish
• Persian
• Finnish
• French
• Hebrew
• Hindi
• Hungarian
• Indonesian
• Italian
• Japanese
• Korean
• Malay
• Dutch
• Norwegian
• Polish
• Pashto
• Portuguese
• Russian
• Swedish
• Simplified Chinese
• Traditional Chinese
• Cantonese
• Tagalog (including Taglish)
• Thai
• Turkish
• Urdu
• Vietnamese

Integrate with the API

In summary, the POST /parse method has 3 attributes: content, language, and settings. All 3 attributes are mandatory.

For example:
{"language": "en", "content": "hello", "settings": {}}

Read on for more info on the response and the settings specs. The method doc pages contain snippets of code for your favorite languages and platforms.

Response Reference

The response of the POST /parse method contains several sections displayed or hidden according to the settings provided.

The common attributes are:

• text (string) - the original input
• reduced_output (boolean) - if the input is too big, and verbose information like the lexical chunk was requested, the verbose information will not be generated, and this flag will be set to true and returned as part of the response
• sentiment (floating-point number) - a number in range -1 to 1 indicating the document-level sentiment. Only shown when document_sentiment setting is set to true.
• signal2noise (floating-point number) - a signal to noise ranking of the text, in relation to the array of concepts specified in the relevant setting. Only shown when the relevant setting exists.

Abusive or Problematic Content

The abuse section is an array of detected instances of content that may violate some terms of use. NOTE: the terms of use in online communities may vary, and so it is up to the administrators to determine whether the content is indeed abusive. For instance, it makes no sense to restrict sexual advances in a dating community, or censor profanities when it's accepted in the bulk of the community.

The section exists if instances of abuse are detected and the abuse setting is either omitted or set to true.

Every instance contains the following attributes:

• offset (unsigned integer) - zero-based offset where the instance starts
• length (unsigned integer) - length of the content
• sentence_index (unsigned integer) - zero-based index of the sentence containing the instance
• text (string) - fragment of text containing the instance (only included if the snippets setting is set to true)
• tags (array of strings) - when exists, provides additional detail about the abuse. For instance, if the fragment is classified as an attempt to sell hard drugs, one of the tags will be _harddrug.
• type (string) - the type of the abuse
• severity (string) - how severe the abuse is. The levels of severity are low, medium, high, and extreme
• explanation (string) - when available, provides rationale for the annotation; set the explain setting to true to enable.

The currently supported types are:

• personal_attack - an insult / attack on the addressee, e.g. an instance of cyberbullying. Please note that an attack on a post or a point, or just negative sentiment is not the same as an insult. The line may be blurred at times. See our Knowledge Base for more information.
• bigotry - hate speech aimed at one of the protected classes. The hate speech detected is not just racial slurs, but, generally, hostile statements aimed at the group as a whole
• profanity - profane language, regardless of the intent
• sexual_advances - welcome or unwelcome attempts to gain some sort of sexual favor or gratification
• criminal_activity - attempts to sell or procure restricted items, criminal services, issuing death threats, and so on
• external_contact - attempts to establish contact or payment via external means of communication, e.g. phone, email, instant messaging (may violate the rules in certain communities, e.g. gig economy portals, e-commerce portals)
• adult_only - activities restricted for minors (e.g. consumption of alcohol)
• mental_issues - content indicative of suicidal thoughts or depression
• allegation - claimed knowledge or accusation of a misconduct (not necessarily crime)
• provocation - content likely to provoke an individual or a group
• disturbing - graphic descriptions that may disturb readers
• no_meaningful_content - unparseable gibberish without apparent meaning
• data_leak - private data like passwords, ID numbers, etc.
• spam - (RESERVED) spam content
• generic - undefined

Sentiment Analysis

The sentiment_expressions section is an array of detected fragments indicating the attitude towards aspects or entities.

The section exists if sentiment is detected and the sentiment setting is either omitted or set to true.

Every instance contains the following attributes:

• offset (unsigned integer) - zero-based offset where the instance starts
• length (unsigned integer) - length of the content
• sentence_index (unsigned integer) - zero-based index of the sentence containing the instance
• text (string) - fragment of text containing the instance (only included if the snippets setting is set to true)
• polarity (string) - whether the attitude is positive, negative, or mixed. Additionally, there is a default sentiment used for cases when the entire snippet has been pre-classified. For instance, if a review is split into two portions, What did you like? and What did you not like?, and the reviewer replies briefly, e.g. The quiet. The service, the utterance itself has no sentiment value. When the calling application is aware of the intended sentiment, the default sentiment simply provides the targets / aspects, which will be then added the sentiment externally.
• targets (array of strings) - when available, provides set of aspects and/or entities which are the targets of the sentiment. For instance, when the utterance is, The breakfast was yummy but the staff is unfriendly, the targets for the two sentiment expressions are meal and staff. Named entities may also be targets of the sentiment.
• reasons (array of strings) - when available, provides reasons for the sentiment. In the example utterance above (The breakfast was yummy but the staff is unfriendly), the reasons array for the staff is ["unfriendly"], while the reasons array for meal is ["tasty"].
• explanation (string) - when available, provides rationale for the sentiment; set the explain setting to true to enable.

Example:

"sentiment_expressions": [
        {
            "sentence_index": 0,
             "offset": 0,
             "length": 32,
             "polarity": "positive",
             "reasons": ["close"],
             "targets": ["location"]
         },
         {
            "sentence_index": 0,
             "offset": 38,
             "length": 29,
             "polarity": "negative",
             "reasons": ["disrespectful"],
             "targets": ["staff"]
         }
     ]

Entities

The entities_summary section is an array of named entity objects detected in the text.

The section exists if named entities are detected and the entities setting is either omitted or set to true.

Every entity contains the following attributes:

• name (string) - the most complete name of the entity in the text of all the mentions
• ref_lemma (string) - when available, the dictionary form of the entity in the reference language (English) regardless of the input language
• type (string) - a string or an array of strings specifying the type of the entity, such as person, organization, numeric, amount_of_money, place. Certain entities, like countries, may have several types (because a country is both a place and an organization).
• subtype (string) - a string indicating the subtype of the entity
• mentions (array of objects) - a set of instances where the entity was mentioned in the text

Every mention contains the following attributes:

• offset (unsigned integer) - zero-based offset where the instance starts
• length (unsigned integer) - length of the content
• sentence_index (unsigned integer) - zero-based index of the sentence containing the instance
• text (string) - fragment of text containing the instance (only included if the snippets setting is set to true)

Example:

 "entities_summary": [
        {
            "type": "person",
             "name": "John Smith",
             "ref_lemma": "John Smith",
             "mentions": [
                {
                    "sentence_index": 0,
                     "offset": 0,
                     "length": 10 }
             ]
         }
    ,
         {
            "type": [ "organization", "place" ]
        ,
             "name": "UK",
             "ref_lemma": "U.K.",
             "mentions": [
                {
                    "sentence_index": 0,
                     "offset": 40,
                     "length": 2 }
             ]
         }
     ]

Entity Types and Subtypes

The currently supported entity types are:

• person, with optional subtypes: fictional_character, important_person, spiritual_being
• organization (note that a country is both an organization and a place)
• place
• time_range
• date
• time
• hashtag
• email
• amount_of_money
• phone phone number, either domestic or international, in a variety of formats
• role (a social role, e.g. position in an organization)
• software
• website (URL), with an optional subtype: tor for Onion links; note that web services may also have the software type assigned
• weight
• bank_account only IBAN format is supported; subtypes: iban
• credit_card, with optional subtypes: visa, mastercard, american_express, diners_club, discovery, jcb, unionpay
• coordinates (GPS coordinates)
• credential, with optional subtypes: md5, sha-1
• crypto, with optional subtypes: bitcoin, ethereum, monero, monero_payment_id, litecoin, dash
• event
• file only Windows pathnames are supported; subtypes: windows, facebook (for images downloaded from Facebook)
• flight_code
• identifier
• ip_address, subtypes: v4, v6
• mac_address
• numeric (an unclassified numeric entity)
• username

Topics

The topics section is an array of topics (subjects, domains, themes in other terms) detected in the text.

The section exists if topics are detected and the topics setting is either omitted or set to true.

By default, a topic is a string. If topic_stats setting is set to true, then every entry in the array contains:

• topic (string) - the topic itself
• coverage (floating-point number) - a number between 0 and 1, indicating the ratio between the number of sentences where the topic is detected to the total number of sentences

Long-Term Memory

The memory section contains optional context to pass to the settings in subsequent messages in the same conversation thread. See Context and Long-Term Memory for more details.

Low-Level: Sentences, Phrases, and Words

Tisane allows obtaining more in-depth data, specifically:

• sentences and their corrected form, if a misspelling was detected
• lexical chunks and their grammatical and stylistic features
• parse trees and phrases

The sentence_list section is generated if the words or the parses setting is set to true.

Every sentence structure in the list contains:

• offset (unsigned integer) - zero-based offset where the sentence starts
• length (unsigned integer) - length of the sentence
• text (string) - the sentence itself
• corrected_text (string) - if a misspelling was detected and the spellchecking is active, contains the automatically corrected text
• words (array of structures) - if words setting is set to true, generates extended information about every lexical chunk. (The term "word" is used for the sake of simplicity, however, it may not be linguistically correct to equate lexical chunks with words.)
• parse_tree (object) - if parses setting is set to true, generates information about the parse tree and the phrases detected in the sentence.
• nbest_parses (array of parse objects) if parses setting is set to true and deterministic setting is set to false, generates information about the parse trees that were deemed close enough to the best one but not the best.

Words

Every lexical chunk ("word") structure in the words array contains:

• type (string) - the type of the element: punctuation for punctuation marks, numeral for numerals, or word for everything else
• text (string) - the text
• offset (unsigned integer) - zero-based offset where the element starts
• length (unsigned integer) - length of the element
• corrected_text (string) - if a misspelling is detected, the corrected form
• lettercase (string) - the original letter case: upper, capitalized, or mixed. If lowercase or no case, the attribute is omitted.
• stopword (boolean) - determines whether the word is a stopword
• grammar (array of strings or structures) - generates the list of grammar features associated with the word. If the feature_standard setting is defined as native, then every feature is an object containing a numeral (index) and a string (value). Otherwise, every feature is a plain string

Advanced

For lexical words only:

• role (string) - semantic role, like agent or patient. Note that in passive voice, the semantic roles are reverse to the syntactic roles. E.g. in a sentence like The car was driven by David, car is the patient, and David is the agent.
• numeric_value (floating-point number) - the numeric value, if the chunk has a value associated with it
• family (integer number) - the ID of the family associated with the disambiguated word-sense of the lexical chunk
• definition (string) - the definition of the family, if the fetch_definitions setting is set to true
• lexeme (integer number) - the ID of the lexeme entry associated with the disambiguated word-sense of the lexical chunk
• nondictionary_pattern (integer number) - the ID of a non-dictionary pattern that matched, if the word was not in the language model but was classified by the nondictionary heuristics
• style (array of strings or structures) - generates the list of style features associated with the word. Only if the feature_standard setting is set to native or description
• semantics (array of strings or structures) - generates the list of semantic features associated with the word. Only if the feature_standard setting is set to native or description
• segmentation (structure) - generates info about the selected segmentation, if there are several possibilities to segment the current lexical chunk and the deterministic setting is set to false. A segmentation is simply an array of word structures.
• other_segmentations (array of structures) - generates info about the segmentations deemed incorrect during the disambiguation process. Every entry has the same structure as the segmentation structure.
• nbest_senses (array of structures) - when the deterministic setting is set to false, generates a set of hypotheses that were deemed incorrect by the disambiguation process. Every hypothesis contains the following attributes: grammar, style, and semantics, identical in structure to their counterparts above; and senses, an array of word-senses associated with every hypothesis. Every sense has a family, which is an ID of the associated family; and, if the fetch_definitions setting is set to true, definition and ref_lemma of that family.

For punctuation marks only:

• id (integer number) - the ID of the punctuation mark
• behavior (string) - the behavior code of the punctuation mark. Values: sentenceTerminator, genericComma, bracketStart, bracketEnd, scopeDelimiter, hyphen, quoteStart, quoteEnd, listComma (for East-Asian enumeration commas like 、)

Parse Trees and Phrases

Every parse tree, or more accurately, parse forest, is a collection of phrases, hierarchically linked to each other.

At the top level of the parse, there is an array of root phrases under the phrases element and the numeric id associated with it. Every phrase may have children phrases. Every phrase has the following attributes:

• type (string) - a Penn treebank phrase tag denoting the type of the phrase, e.g. S, VP, NP, etc.
• family (integer number) - an ID of the phrase family
• offset (unsigned integer) - a zero-based offset where the phrase starts
• length (unsigned integer) - the span of the phrase
• role (string) - the semantic role of the phrase, if any, analogous to that of the words
• text (string) - the phrase text, where the phrase members are delimited by the vertical bar character. Children phrases are enclosed in brackets. E.g., driven|by|David or (The|car)|was|(driven|by|David).

Example:

"parse_tree": {
"id": 4,
"phrases": [
{
        "type": "S",
        "family": 1451,
        "offset": 0,
        "length": 27,
        "text": "(The|car)|was|(driven|by|David)",
        "children": [
                {
                        "type": "NP",
                        "family": 1081,
                        "offset": 0,
                        "length": 7,
                        "text": "The|car",
                        "role": "patient"
                },
                {
                        "type": "VP",
                        "family": 1172,
                        "offset": 12,
                        "length": 15,
                        "text": "driven|by|David",
                        "role": "verb"
                }
        ]
}

Context-Aware Spelling Correction

Tisane supports automatic, context-aware spelling correction. Whether it's a misspelling or a purported obfuscation, Tisane attempts to deduce the intended meaning, if the language model does not recognize the word.

When or if it's found, Tisane adds the corrected_text attribute to the word (if the words / lexical chunks are returned) and the sentence (if the sentence text is generated). Sentence-level corrected_text is displayed if words or parses are set to true.

Note that as Tisane works with large dictionaries, you may need to exclude more esoteric terms by using the min_generic_frequency setting.

Note that the invocation of spell-checking does not depend on whether the sentences and the words sections are generated in the output. The spellchecking can be disabled by setting disable_spellcheck to true. Another option is to enable the spellchecking for lowercase words only, thus excluding potential proper nouns in languages that support capitalization; to avoid spell-checking capitalized and uppercase words, set lowercase_spellcheck_only to true.

Settings Reference

The purpose of the settings structure is to:

• provide cues about the content being sent to improve the results
• customize the output and select sections to be shown
• define standards and formats in use
• define and calculate the signal to noise ranking

All settings are optional. To leave all settings to default, simply provide an empty object ({}).

Content Cues and Instructions

format (string) - the format of the content. Some policies will be applied depending on the format. Certain logic in the underlying language models may require the content to be of a certain format (e.g. logic applied on the reviews may seek for sentiment more aggressively). The default format is empty / undefined. The format values are:

• review - a review of a product or a service or any other review. Normally, the underlying language models will seek for sentiment expressions more aggressively in reviews.
• dialogue - a comment or a post which is a part of a dialogue. An example of a logic more specific to a dialogue is name calling. A single word like "idiot" would not be a personal attack in any other format, but it is certainly a personal attack when part of a dialogue.
• shortpost - a microblogging post, e.g. a tweet.
• longform - a long post or an article.
• proofread - a post which was proofread. In the proofread posts, the spellchecking is switched off.
• alias - a nickname in an online community.
• search - a search query. Search queries may not always be grammatically correct. Certain topics and items, that we may otherwise let pass, are tagged with the search format.

disable_spellcheck (boolean) - determines whether the automatic spellchecking is to be disabled. Default: false.

lowercase_spellcheck_only (boolean) - determines whether the automatic spellchecking is only to be applied to words in lowercase. Default: false

min_generic_frequency (int) - allows excluding more esoteric terms; the valid values are 0 thru 10.

subscope (boolean) - enables sub-scope parsing, for scenarios like hashtag, URL parsing, and obfuscated content (e.g. ihateyou). Default: false.

lang_detect_segmentation_regex (string) - allows defining custom language detection fragment boundaries. For example, if multiple languages may be used in different sentences in the same text, you may want to define the regex as: (([\r\n]|[.!?][ ])) .

domain_factors (set of pairs made of strings and numbers) - provides a session-scope cues for the domains of discourse. This is a powerful tool that allows tailoring the result based on the use case. The format is, family ID of the domain as a key and the multiplication factor as a value (e.g. *"12345": 5.0*). For example, when processing text looking for criminal activity, we may want to set domains relevant to drugs, firearms, crime, higher: "domain_factors": {"31058": 5.0, "45220": 5.0, "14112": 5.0, "14509": 3.0, "28309": 5.0, "43220": 5.0, "34581": 5.0}. The same device can be used to eliminate noise coming from domains we know are irrelevant by setting the factor to a value lower than 1.

when (date string, format YYYY-MM-DD) - indicates when the utterance was uttered. (TO BE IMPLEMENTED) The purpose is to prune word senses that were not available at a particular point in time. For example, the words troll, mail, and post had nothing to do with the Internet 300 years ago because there was no Internet, and so in a text that was written hundreds of years ago, we should ignore the word senses that emerged only recently.

Output Customization

abuse (boolean) - output instances of abusive content (default: true)

sentiment (boolean) - output sentiment-bearing snippets (default: true)

document_sentiment (boolean) - output document-level sentiment (default: false)

entities (boolean) - output entities (default: true)

topics (boolean) - output topics (default: true), with two more relevant settings:

• topic_stats (boolean) - include coverage statistics in the topic output (default: false). When set, the topic is an object containing the attributes topic (string) and coverage (floating-point number). The coverage indicates a share of sentences touching the topic among all the sentences.
• optimize_topics (boolean) - if true, the less specific topics are removed if they are parts of the more specific topics. For example, when the topic is cryptocurrency, the optimization removes finance.

words (boolean) - output the lexical chunks / words for every sentence (default: false). In languages without white spaces (Chinese, Japanese, Thai), the tokens are tokenized words. In languages with compounds (e.g. German, Dutch, Norwegian), the compounds are split.

fetch_definitions (boolean) - include definitions of the words in the output (default: false). Only relevant when the words setting is true

parses (boolean) - output parse forests of phrases

deterministic (boolean) - whether the n-best senses and n-best parses are to be output in addition to the detected sense. If true, only the detected sense will be output. Default: true

snippets (boolean) - include the text snippets in the abuse, sentiment, and entities sections (default: false)

explain (boolean) - if true, a reasoning for the abuse and sentiment snippets is provided when possible (see the explanation attribute)

Standards and Formats

feature_standard (string) - determines the standard used to output the features (grammar, style, semantics) in the response object. The standards we support are:

• ud: Universal Dependencies tags (default)
• penn: Penn treebank tags
• native: Tisane native feature codes
• description: Tisane native feature descriptions

Only the native Tisane standards (codes and descriptions) support style and semantic features.

topic_standard (string) - determines the standard used to output the topics in the response object. The standards we support are:

• iptc_code - IPTC topic taxonomy code
• iptc_description - IPTC topic taxonomy description
• iab_code - IAB topic taxonomy code
• iab_description - IAB topic taxonomy description
• native - Tisane domain description, coming from the family description (default)

sentiment_analysis_type (string) - the type of the sentiment analysis strategy. The values are:

• products_and_services - most common sentiment analysis of products and services
• entity - sentiment analysis with entities as targets
• creative_content_review - reviews of creative content (RESERVED)
• political_essay - political essays (RESERVED)

Context and Long-Term Memory

Human understanding of language is not a simple "sliding window" with scope limited to a sentence. Language is accompanied by gestures, visuals, and knowledge of the previous communication. Sometimes, code-words may be used to conceal the words' original meaning.

When detecting abuse, a name of an ethnicity or a religious group may be not offensive, but when superimposed over a picture of an ape or a pig, it is meant of offend. When translating from a language without gender distinctions in verbs (like English) to a language with distinctions (like Russian or Hebrew), there is no way to know from an utterance alone if the speaker is female. When a scammer is collecting details piecemeal over a series of utterances, knowledge of previous utterances is needed to take action.

Tisane's Memory module allows pre-initializing the analysis, as well as reassigning meanings, and more. The module is made of three simple components that are flexible enough for a variety of tasks:

Reassignments

Reassignments define the attributes to set based on other attributes. This allows to:

• assign gender to 1st or 2nd person verbs, generating accurate translations
• overwrite original meaning of a group of words with all their inflected forms to analyze code-words and secret language
• add an additional feature or a hypernym to a family

and more, within a scope of a call.

The assign section is an array of structures defining:

• if - conditions to match:
  • regex - a regular expression (RE2 syntax)
  • family - a family ID
  • features - a list of feature values. A feature is a structure with an index and a value. For example: {"index":1, "value":"NOUN"}.
  • hypernym - a family ID of a hypernym
• then - attributes to assign
  • family - a family ID
  • features - a list of feature values. A feature is a structure with an index and a value. For example: {"index":1, "value":"NOUN"}.
  • hypernym - a family ID of a hypernym

Examples:

• the speaker is female: `"assign":[{"if":{"features":[{"index":9,"value":"1"}]},"then":{"features":[{"index":5,"value":"F"}]}}]
• assume that a mention of a container refers to an illegal item: `"assign":[{"if":{"family":26888},"then":{"hypernym":123078}}]

Flags

An array of flag structures that add some context. A flag is a structure with an index and a value. For example: {"index":36, "value":"WFH"}.

Aside from the flags returned in the memory section of the response, these flags can be set:

• {"index":36, "value":"PEBD"} (agents_of_bad_things) - the context is about a bad player or an agent responsible for bad things
• {"index":36, "value":"BADANML"} (bad_animal) - the context is an animal that symbolizes bad qualities (e.g. pig, ape, snake, etc.)
• {"index":36, "value":"BULKMSG"} (bulk_message) - the message was sent in bulk
• {"index":36, "value":"DETHR"} (death_related) - the context is something related to death
• {"index":36, "value":"EARNMUCH"} (make_money) - the context is related to making money
• {"index":36, "value":"IDEP"} (my_departure) - the author of the text mentioned departing
• {"index":36, "value":"SECO"} (sexually_conservative) - any attempt to exchange photos or anything that may be either sexual or non-sexual is to be deemed sexual
• {"index":36, "value":"TRPA"} (trusted_party) - the author of the text claims to be a trusted party (e.g. a relative or a spouse)
• {"index":36, "value":"WSTE"} (waste) - the context is about waste, organic or inorganic
• {"index":36, "value":"WOPR"} (won_prize) - prize or money winning was mentioned or implied
• {"index":36, "value":"WFH"} (work_from_home) - work from home was mentioned
• {"index":5, "value":"ORG"} (organization) - an organization was mentioned
• {"index":5, "value":"ROLE"} (role) - a role or a position was mentioned

Antecedents

The section contains structures to be used in coreference resolution. The attributes are:

• family - the family ID of the antecedent
• features - the list of features. Every feature is a structure with an index and a value. For example: {"index":36, "value":"WFH"}.

Signal to Noise Ranking

When we're studying a bunch of posts commenting on an issue or an article, we may want to prioritize the ones more relevant to the topic, and containing more reason and logic than emotion. This is what the signal to noise ranking is meant to achieve.

The signal to noise ranking is made of two parts:

1. Determine the most relevant concepts. This part may be omitted, depending on the use case scenario (e.g. we want to track posts most relevant to a particular set of issues).
2. Rank the actual post in relevance to these concepts.

To determine the most relevant concepts, we need to analyze the headline or the article itself. The headline is usually enough. We need two additional settings:

• keyword_features (an object of strings with string values) - determines the features to look for in a word. When such a feature is found, the family ID is added to the set of potentially relevant family IDs.
• stop_hypernyms (an array of integers) - if a potentially relevant family ID has a hypernym listed in this setting, it will not be considered. For example, we extracted a set of nouns from the headline, but we may not be interested in abstractions or feelings. E.g. from a headline like Fear and Loathing in Las Vegas we want Las Vegas only. Optional.

If keyword_features is provided in the settings, the response will have a special attribute, relevant, containing a set of family IDs.

At the second stage, when ranking the actual posts or comments for relevance, this array is to be supplied among the settings. The ranking is boosted when the domain, the hypernyms, or the families related to those in the relevant array are mentioned, when negative and positive sentiment is linked to aspects, and penalized when the negativity is not linked to aspects, or abuse of any kind is found. The latter consideration may be disabled, e.g. when we are looking for specific criminal content. When the abuse_not_noise parameter is specified and set to true, the abuse is not penalized by the ranking calculations.

To sum it up, in order to calculate the signal to noise ranking:

1. Analyze the headline with keyword_features and, optionally, stop_hypernyms in the settings. Obtain the relevant attribute.
2. When analyzing the posts or the comments, specify the relevant attribute obtained in step 1.