Accuracy
A score is only worth what it has been checked against. Here is what ours have been checked against, and, as plainly, what has not been checked yet.
Adjectives
Measured against English annotated word by word by linguists (Universal Dependencies), on text the detector never trained on.
| Test text | Accuracy (F1) |
|---|---|
| Web text (reviews, email) | 0.87 |
| Mixed genres | 0.90 |
| News only | 0.84 (finds 94% of adjectives) |
F1 combines how many adjectives it finds and how many of its finds are right; 1.0 is perfect.
Our version is always calmer
Measured on every article we check, our version never scores higher than the original on the Niral Score, Happiness, sentiment strength, political, generational, gender, jingoistic or religious wording, or adjective count (checked on 400 articles and on every test fixture, on every change).
Against people
The indicators below have been built and tested against examples we wrote, which is not the same as agreeing with readers. We are collecting labels from people to measure that, and publish the results here as they come in.
| Indicator | Status |
|---|---|
| Adjectives, as readers see them | Not yet measured 0 of 30 paragraphs labelled. |
| Niral Score vs how loaded people find it | Not yet measured 0 of 30 paragraphs labelled. |
| Sentiment vs tone people read | Not yet measured 0 of 30 paragraphs labelled. |
| Drama (Happiness) vs "sensational" | Not yet measured 0 of 30 paragraphs labelled. |
| Political, gender, generational, jingoistic, religious | Not yet measured Planned next: 300 Australian articles labelled by at least two people each. |
0 labels from 0 people so far.
Help measure it
It takes a minute a paragraph: mark the describing words and say how loaded it reads. Label a paragraph (a free account, so each person is counted once).