19 lines
8.4 KiB
JSON
19 lines
8.4 KiB
JSON
# Failure-mode judge calibration set
|
|
# brief: PM-Waggle-OS/briefs/2026-04-20-cc-preflight-prep-tasks.md Task 2
|
|
# locked: decisions/2026-04-20-failure-mode-oq-resolutions-locked.md §OQ-FM-3
|
|
# source: https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json
|
|
# seed=43 (non-overlapping with preflight-locomo-50.json seed=42)
|
|
# distribution: single-hop=3, multi-hop=3, temporal=2, open-ended=2 (total=10)
|
|
# human_label.{verdict,failure_mode,rationale} are left null. PM labels first pass; CC validates second pass.
|
|
# Judge activates Stage 1 only after ≥8/10 match against human_label.
|
|
{"id":"locomo_conv-26_q109","category":"single-hop","context":"Session 8 (1:51 pm on 15 July, 2023):\nMelanie: Hey Caroline, it's been super busy here. So much since we talked! Last Fri I finally took my kids to a pottery workshop. We all made our own pots, it was fun and therapeutic! [image: a photography of a group of children making clay sculptures in a classroom]","question":"What did Mel and her kids make during the pottery workshop?","ground_truth_answer":"pots","locomo_metadata":{"sample_id":"conv-26","qa_index":109,"locomo_category":4,"evidence":["D8:2"],"speaker_a":"Caroline","speaker_b":"Melanie"},"human_label":{"verdict":null,"failure_mode":null,"rationale":null}}
|
|
{"id":"locomo_conv-41_q123","category":"single-hop","context":"Session 23 (6:29 pm on 7 July, 2023):\nMaria: For sure! We can keep doing great stuff and making a difference. Well, I'm off to have dinner with some friends from the gym. Talk to you later!","question":"What did Maria plan to do later on the evening of 7 July, 2023?","ground_truth_answer":"have dinner with friends from the gym","locomo_metadata":{"sample_id":"conv-41","qa_index":123,"locomo_category":4,"evidence":["D23:14"],"speaker_a":"John","speaker_b":"Maria"},"human_label":{"verdict":null,"failure_mode":null,"rationale":null}}
|
|
{"id":"locomo_conv-50_q141","category":"single-hop","context":"Session 27 (10:49 am on 29 October, 2023):\nDave: That's Boston, Cal! Check this out, I took this picture last month, and got a great shot - it was stunning! [image: a photography of a clock tower in a city with buildings]","question":"Which city is featured in the photograph Dave showed Calvin?","ground_truth_answer":"Boston","locomo_metadata":{"sample_id":"conv-50","qa_index":141,"locomo_category":4,"evidence":["D27:6"],"speaker_a":"Calvin","speaker_b":"Dave"},"human_label":{"verdict":null,"failure_mode":null,"rationale":null}}
|
|
{"id":"locomo_conv-42_q030","category":"multi-hop","context":"Session 2 (2:01 pm on 23 January, 2022):\nJoanna: Woo! I finally finished my first full screenplay and printed it last Friday. I've been working on for a while, such a relief to have it all done! [image: a photography of a book with a page of text on it]\n\nSession 17 (2:34 pm on 10 July, 2022):\nJoanna: I will! I actually started on a book recently since my movie did well! [image: a photo of a person holding a notebook with a handwritten page]\n\nSession 18 (6:12 pm on 14 August, 2022):\nJoanna: Hey Nate, long time no talk! I've been busy with writing projects and really going all out with it. It's been the best thing ever - a mix of highs and lows - and my journal's pretty much my rock. Writing's such a huge part of me now. [image: a photo of a notebook with a bunch of stickers on it]\nJoanna: Yep. Last week, someone wrote me a letter after reading an online blog post I made about a hard moment in my life. Their words touched me; they said my story had brought them comfort. It was awesome to realize my words had that kind of power. It reminded me why I love writing. [image: a photo of a note written to a person on a piece of paper]","question":"What kind of writings does Joanna do?","ground_truth_answer":"Screenplays,books, online blog posts, journal","locomo_metadata":{"sample_id":"conv-42","qa_index":30,"locomo_category":1,"evidence":["D2:3","D17:14","D18:1","D18:5"],"speaker_a":"Joanna","speaker_b":"Nate"},"human_label":{"verdict":null,"failure_mode":null,"rationale":null}}
|
|
{"id":"locomo_conv-49_q015","category":"multi-hop","context":"Session 3 (3:55 pm on 6 June, 2023):\nSam: Nah, no changes for me. Still enjoying my soda and candy, although I know it's not the best habit to have.","question":"What kind of unhealthy snacks does Sam enjoy eating?","ground_truth_answer":"soda, candy","locomo_metadata":{"sample_id":"conv-49","qa_index":15,"locomo_category":1,"evidence":["D3:4"],"speaker_a":"Evan","speaker_b":"Sam"},"human_label":{"verdict":null,"failure_mode":null,"rationale":null}}
|
|
{"id":"locomo_conv-41_q036","category":"multi-hop","context":"Session 8 (6:03 pm on 6 March, 2023):\nJohn: I just try to find things that we'll have fun with, like a walk or picnic in the park, or finding events in our town and beyond. Just last week, I found a violin concert that we all enjoyed. It's all about making memories together.\n\nSession 20 (12:21 am on 27 June, 2023):\nJohn: That's great, Maria! It's such a blessing to have family who always supports us and reminds us that we're not alone. They know us like no one else and stick by us no matter what. Last week, we had a blast at a live music event. Seeing them dancing and having fun was awesome. The energy in the air was amazing.","question":"What music events has John attended?","ground_truth_answer":"Live music event, violin concert","locomo_metadata":{"sample_id":"conv-41","qa_index":36,"locomo_category":1,"evidence":["D20:4","D8:12"],"speaker_a":"John","speaker_b":"Maria"},"human_label":{"verdict":null,"failure_mode":null,"rationale":null}}
|
|
{"id":"locomo_conv-42_q038","category":"temporal","context":"Session 21 (1:43 pm on 14 September, 2022):\nJoanna: Cool, Nate! Gonna give it a go. Dairy-free is a must for me, especially for desserts. Last Friday, I made a deeeelish dessert with almond milk - it was good! Got any favs when it comes to dairy-free desserts?","question":"When did Joanna make a desert with almond milk?","ground_truth_answer":"The Friday before 14September, 2022","locomo_metadata":{"sample_id":"conv-42","qa_index":38,"locomo_category":2,"evidence":["D21:9"],"speaker_a":"Joanna","speaker_b":"Nate"},"human_label":{"verdict":null,"failure_mode":null,"rationale":null}}
|
|
{"id":"locomo_conv-41_q053","category":"temporal","context":"Session 28 (5:19 pm on 5 August, 2023):\nJohn: I haven't been able to volunteer much lately, but I definitely care about it. Last year, I helped renovate a rundown community center back home. It took a lot of work, but seeing the impact on the community was so worth it.","question":"When did John help renovate his hometown community center?","ground_truth_answer":"2022","locomo_metadata":{"sample_id":"conv-41","qa_index":53,"locomo_category":2,"evidence":["D28:11"],"speaker_a":"John","speaker_b":"Maria"},"human_label":{"verdict":null,"failure_mode":null,"rationale":null}}
|
|
{"id":"locomo_conv-50_q037","category":"open-ended","context":"Session 7 (6:06 pm on 31 May, 2023):\nCalvin: Hey Dave! Been ages since we chatted. So much has gone down. Touring with Frank Ocean last week was wild. Tokyo was unreal -- the crowd was insane. I felt so alive when performing, can't find the words to express it.\n\nSession 16 (2:55 pm on 31 August, 2023):\nCalvin: Hey Dave! The tour was amazing! I was so pumped from all the energy from the audience. This pic totally captures how I felt. Such an amazing time! [image: a photo of a crowd of people watching a concert with a large screen]\n\nSession 18 (10:56 am on 13 September, 2023):\nCalvin: Thanks Dave! Lots of cool stuff happening. Next up, a tour - so excited! After that, I'm off to explore and grow my brand. Can't wait to see what the future has in store!","question":"Does Calvin love music tours?","ground_truth_answer":"yes","locomo_metadata":{"sample_id":"conv-50","qa_index":37,"locomo_category":3,"evidence":["D18:7","D16:2","D7:1"],"speaker_a":"Calvin","speaker_b":"Dave"},"human_label":{"verdict":null,"failure_mode":null,"rationale":null}}
|
|
{"id":"locomo_conv-47_q017","category":"open-ended","context":"Session 8 (2:36 pm on 29 April, 2022):\nJohn: Yeah for sure! I've been playing one more game with friends these days. It's a game to figure out who the impostors are and it's super fun.","question":"What is the board game where you have to find the imposter that John mentions to James?","ground_truth_answer":"Mafia","locomo_metadata":{"sample_id":"conv-47","qa_index":17,"locomo_category":3,"evidence":["D8:36"],"speaker_a":"James","speaker_b":"John"},"human_label":{"verdict":null,"failure_mode":null,"rationale":null}}
|