Context-sensitive Fine-grained Lexical Inference Dataset

A dataset for fine-grained lexical inference in-context, as described in:

Adding Context to Semantic Data-Driven Paraphrasing
Vered Shwartz and Ido Dagan. *SEM 2016. [paper]

The dataset is available here.

The filtered version of this dataset contains 91% of the entries, which are those that at least 3 out of 5 annotators agreed on their labels. It is available here.

The annotation guidelines and AMT templates are available here.

Published Results:

Publication Model Precision Recall F1
Shwartz and Dagan, 2016 PPDB-fine-human manual annotations (out-of-context) 0.722 0.380 0.288
Shwartz and Dagan, 2016 PPDB 2.0 classifier predictions (out-of-context) 0.611 0.565 0.556
Shwartz and Dagan, 2016 LR classifier with PPDB 2.0 features + context-sensitive features 0.677 0.685 0.670

If you have new results on this table, please email me to and I will include them here.

In Archive