A dataset for fine-grained lexical inference in-context, as described in:
Adding Context to Semantic Data-Driven Paraphrasing
Vered Shwartz and Ido Dagan. *SEM 2016. [paper]
The dataset is available here.
The filtered version of this dataset contains 91% of the entries, which are those that at least 3 out of 5 annotators agreed on their labels. It is available here.
The annotation guidelines and AMT templates are available here.
Publication Model Precision Recall F1 Shwartz and Dagan, 2016 PPDB-fine-human manual annotations (out-of-context) 0.722 0.380 0.288 Shwartz and Dagan, 2016 PPDB 2.0 classifier predictions (out-of-context) 0.611 0.565 0.556 Shwartz and Dagan, 2016 LR classifier with PPDB 2.0 features + context-sensitive features 0.677 0.685 0.670
If you have new results on this table, please email me to firstname.lastname@example.org and I will include them here.