8 lines
5.4 KiB
JSON
8 lines
5.4 KiB
JSON
{"instance_id": "locomo_conv-43_q155", "cell": "agentic", "question": "How did John feel after being able to jog without pain?", "ground_truth": "It was a huge success.", "context": "Session 24 (3:37 pm on 16 December, 2023):\nJohn: It was great! After being out for so long, jogging without any pain was a huge success. My wife and I hosted a small get-together with friends and family to celebrate.", "model_answer": "great", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict": "correct", "gemini_failure_mode": null}
|
|
{"instance_id": "locomo_conv-50_q015", "cell": "agentic", "question": "Why does Dave regularly visit parks?", "ground_truth": "because it relaxes and calms him", "context": "Session 1 (11:53 am on 23 March, 2023):\nDave: That's great, can't wait for you to return from your trip. I've been spending lots of time at this beautiful park lately - so calming. Have you been there before? [image: a photography of a park with a lake and a few boats]\n\nSession 8 (2:31 pm on 9 June, 2023):\nDave: Hey, nice photo of the studio! Working in a team yields incredible results, well done! Keep pushing it! I've been doing good - thanks for asking. I've been exploring some parks on the weekends to relax - it's so peaceful being surrounded by nature. Are there any chill spots you enjoy in Boston?", "model_answer": "to relax and recharge", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict": "correct", "gemini_failure_mode": null}
|
|
{"instance_id": "locomo_conv-43_q155", "cell": "full-context", "question": "How did John feel after being able to jog without pain?", "ground_truth": "It was a huge success.", "context": "Session 24 (3:37 pm on 16 December, 2023):\nJohn: It was great! After being out for so long, jogging without any pain was a huge success. My wife and I hosted a small get-together with friends and family to celebrate.", "model_answer": "great", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict": "correct", "gemini_failure_mode": null}
|
|
{"instance_id": "locomo_conv-50_q015", "cell": "full-context", "question": "Why does Dave regularly visit parks?", "ground_truth": "because it relaxes and calms him", "context": "Session 1 (11:53 am on 23 March, 2023):\nDave: That's great, can't wait for you to return from your trip. I've been spending lots of time at this beautiful park lately - so calming. Have you been there before? [image: a photography of a park with a lake and a few boats]\n\nSession 8 (2:31 pm on 9 June, 2023):\nDave: Hey, nice photo of the studio! Working in a team yields incredible results, well done! Keep pushing it! I've been doing good - thanks for asking. I've been exploring some parks on the weekends to relax - it's so peaceful being surrounded by nature. Are there any chill spots you enjoy in Boston?", "model_answer": "to relax", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict": "correct", "gemini_failure_mode": null}
|
|
{"instance_id": "locomo_conv-43_q155", "cell": "oracle-context", "question": "How did John feel after being able to jog without pain?", "ground_truth": "It was a huge success.", "context": "Session 24 (3:37 pm on 16 December, 2023):\nJohn: It was great! After being out for so long, jogging without any pain was a huge success. My wife and I hosted a small get-together with friends and family to celebrate.", "model_answer": "Great", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict": "correct", "gemini_failure_mode": null}
|
|
{"instance_id": "locomo_conv-47_q092", "cell": "oracle-context", "question": "What did John create for the charitable foundation that helped generate reports for analysis?", "ground_truth": "computer application on smartphones", "context": "Session 11 (5:00 pm on 11 May, 2022):\nJohn: Previously, this foundation used paper records and all inventory was recorded manually. I made an application that structured their work, and now everything they need for inventory is in one application on their smartphone.", "model_answer": "An application", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict": "incorrect", "gemini_failure_mode": "F2"}
|
|
{"instance_id": "locomo_conv-50_q015", "cell": "oracle-context", "question": "Why does Dave regularly visit parks?", "ground_truth": "because it relaxes and calms him", "context": "Session 1 (11:53 am on 23 March, 2023):\nDave: That's great, can't wait for you to return from your trip. I've been spending lots of time at this beautiful park lately - so calming. Have you been there before? [image: a photography of a park with a lake and a few boats]\n\nSession 8 (2:31 pm on 9 June, 2023):\nDave: Hey, nice photo of the studio! Working in a team yields incredible results, well done! Keep pushing it! I've been doing good - thanks for asking. I've been exploring some parks on the weekends to relax - it's so peaceful being surrounded by nature. Are there any chill spots you enjoy in Boston?", "model_answer": "To relax", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict": "correct", "gemini_failure_mode": null}
|