101 lines
83 KiB
JSON
101 lines
83 KiB
JSON
{"instance_id": "locomo_conv-50_q086", "cell": "no-context", "question": "What sports activity is Calvin planning to try after the tour with Frank Ocean?", "ground_truth": "Skiing", "context": "Session 9 (3:15 pm on 21 June, 2023):\nCalvin: Nope, not yet! I haven't been there before, but it's on my to-do list for after my tour with Frank Ocean ends. Can't wait to see them in person! Look at that snowy peak. [image: a photo of a person on skis on a snowy mountain]", "model_answer": "Surfing", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F3", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
|
||
{"instance_id": "locomo_conv-26_q059", "cell": "no-context", "question": "Would Caroline be considered religious?", "ground_truth": "Somewhat, but not extremely religious", "context": "Session 12 (1:50 pm on 17 August, 2023):\nCaroline: Hey Mel! How're ya doin'? Recently, I had a not-so-great experience on a hike. I ran into a group of religious conservatives who said something that really upset me. It made me think how much work we still have to do for LGBTQ rights. It's been so helpful to have people around me who accept and support me, so I know I'll be ok!\n\nSession 14 (1:33 pm on 25 August, 2023):\nCaroline: Thanks! It was made for a local church and shows time changing our lives. I made it to show my own journey as a transgender woman and how we should accept growth and change. [image: a photo of a large stained glass window in a church]", "model_answer": "Cannot be determined without context.", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
|
||
{"instance_id": "locomo_conv-44_q000", "cell": "no-context", "question": "Which year did Audrey adopt the first three of her dogs?", "ground_truth": "2020", "context": "Session 1 (1:10 pm on 27 March, 2023):\nAudrey: I've had them for 3 years! Their names are Pepper, Precious and Panda. I can't live without my little ones!", "model_answer": "1954", "opus_verdict": "incorrect", "opus_failure_mode": "F4", "gpt_verdict": "incorrect", "gpt_failure_mode": "F4", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
|
||
{"instance_id": "locomo_conv-41_q080", "cell": "no-context", "question": "What did Maria make for her home to remind her of a trip to England?", "ground_truth": "painting of a castle on a hill", "context": "Session 8 (6:03 pm on 6 March, 2023):\nMaria: Thanks, John! I got the idea from that trip to England a few years ago - I was mesmerized by the castles. I wanted to bring that same magic to my home, like a reminder of the world's beauty. I made a painting too! [image: a photography of a painting of a castle on a hill]", "model_answer": "A wind chime", "opus_verdict": "incorrect", "opus_failure_mode": "F4", "gpt_verdict": "incorrect", "gpt_failure_mode": "F3", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
|
||
{"instance_id": "locomo_conv-42_q001", "cell": "no-context", "question": "What kind of interests do Joanna and Nate share?", "ground_truth": "Watching movies, making desserts", "context": "Session 1 (7:31 pm on 21 January, 2022):\nJoanna: Yeah! Besides writing, I also enjoy reading, watching movies, and exploring nature. Anything else you enjoy doing, Nate?\nNate: Playing video games and watching movies are my main hobbies.\nJoanna: Cool, Nate! So we both have similar interests. What type of movies do you like best?\n\nSession 3 (9:27 am on 7 February, 2022):\nNate: Thanks, Joanna. Not much has changed for me, but I just discovered that I can make coconut milk icecream and gave it a try. It was actually pretty good, so I'm proud of myself. [image: a photo of a bowl of ice cream with a spoon in it]\n\nSession 4 (1:07 pm on 25 February, 2022):\nNate: Thanks, Joanna! It means a lot that you enjoy the desserts I bake.\n\nSession 10 (11:54 am on 2 May, 2022):\nJoanna: Not much is new other than the screenplay. Been working on some projects and testing out dairy-free dessert recipes for friends and fam. Here's a pic of a cake I made recently! [image: a photo of a cake with white frosting on a wooden table]\n\nSession 20 (6:03 pm on 5 September, 2022):\nJoanna: Hey Nate! Cute turtles! Bummer about the setback. Any positive vibes comin' your way? I just revised on of my old recipes and made this! [image: a photo of a piece of cake with strawberries and chocolate]", "model_answer": "Reading", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F3", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F3"}
|
||
{"instance_id": "locomo_conv-43_q155", "cell": "no-context", "question": "How did John feel after being able to jog without pain?", "ground_truth": "It was a huge success.", "context": "Session 24 (3:37 pm on 16 December, 2023):\nJohn: It was great! After being out for so long, jogging without any pain was a huge success. My wife and I hosted a small get-together with friends and family to celebrate.", "model_answer": "Relieved.", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F3", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
|
||
{"instance_id": "locomo_conv-50_q015", "cell": "no-context", "question": "Why does Dave regularly visit parks?", "ground_truth": "because it relaxes and calms him", "context": "Session 1 (11:53 am on 23 March, 2023):\nDave: That's great, can't wait for you to return from your trip. I've been spending lots of time at this beautiful park lately - so calming. Have you been there before? [image: a photography of a park with a lake and a few boats]\n\nSession 8 (2:31 pm on 9 June, 2023):\nDave: Hey, nice photo of the studio! Working in a team yields incredible results, well done! Keep pushing it! I've been doing good - thanks for asking. I've been exploring some parks on the weekends to relax - it's so peaceful being surrounded by nature. Are there any chill spots you enjoy in Boston?", "model_answer": "Unknown.", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
|
||
{"instance_id": "locomo_conv-48_q020", "cell": "no-context", "question": "Which games have Jolene and her partner played together?", "ground_truth": "Detroit, Walking Dead, Battlefield 1, It Takes Two, Overcooked 2", "context": "Session 2 (9:49 am on 27 January, 2023):\nJolene: They are very unusual pets! Here's me and my partner gaming last week - it's so fun. We played the game \"Detroit\" on the console. We are both crazy about this activity! [image: a photo of a person laying in bed with a dog watching tv]\nJolene: We are planning to play \"Walking Dead\" next Saturday.\n\nSession 15 (7:37 pm on 9 July, 2023):\nJolene: Yeah, we love playing \"It takes two\" together! It's a fun team-strategy game and it's competitive. Plus, it's a great way for us to bond. Do you have any activities you like doing with people?\n\nSession 19 (12:52 am on 19 August, 2023):\nJolene: Oh, I forgot to mention Overcooked 2 - this is a good co-op game if you're into hilarious and chaotic cooking. My partner and I often play for bets! I once won three large pizzas!\n\nSession 20 (9:11 am on 21 August, 2023):\nJolene: Long time no talk! We were given a new game for the console last week, it is Battlefield 1. What's been up with you?", "model_answer": "Animal Crossing and Minecraft", "opus_verdict": "incorrect", "opus_failure_mode": "F4", "gpt_verdict": "incorrect", "gpt_failure_mode": "F4", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
|
||
{"instance_id": "locomo_conv-48_q055", "cell": "no-context", "question": "Which pet did Jolene adopt more recently - Susie or Seraphim?", "ground_truth": "Seraphim", "context": "Session 2 (9:49 am on 27 January, 2023):\nJolene: I bought it a year ago in Paris.\nJolene: Even as a child I learned to play on my own.\n\nSession 16 (9:26 am on 1 August, 2023):\nJolene: I adopted her two years ago when I was feeling lonely and wanted some company.", "model_answer": "Seraphim", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-26_q128", "cell": "no-context", "question": "What did Caroline find in her neighborhood during her walk?", "ground_truth": "a rainbow sidewalk", "context": "Session 14 (1:33 pm on 25 August, 2023):\nCaroline: I was out walking in my neighborhood when I came across this cool rainbow sidewalk for Pride Month. It was so vibrant and welcoming, I had to take a picture! It reminds us that love and acceptance are everywhere—even where we least expect it.", "model_answer": "A butterfly.", "opus_verdict": "incorrect", "opus_failure_mode": "F4", "gpt_verdict": "incorrect", "gpt_failure_mode": "F4", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
|
||
{"instance_id": "locomo_conv-50_q010", "cell": "no-context", "question": "When did Calvin's place get flooded in Tokyo?", "ground_truth": "On a week before 16 May, 2023", "context": "Session 6 (11:50 am on 16 May, 2023):\nCalvin: Hey Dave, not everything has been going smoothly. I had an incident last week where my place got flooded, but thankfully, I managed to save my music gear and favorite microphone. It's been tough, but I'm staying positive and looking forward to getting everything fixed up.", "model_answer": "October 2019", "opus_verdict": "incorrect", "opus_failure_mode": "F4", "gpt_verdict": "incorrect", "gpt_failure_mode": "F4", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
|
||
{"instance_id": "locomo_conv-50_q078", "cell": "no-context", "question": "What was the necklace Calvin received meant to remind him of?", "ground_truth": "why he keeps hustling as a musician", "context": "Session 4 (6:24 pm on 1 May, 2023):\nCalvin: Thanks, Dave! I got it from another artist as a gift - it's a great reminder of why I keep hustling as a musician!", "model_answer": "Home.", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F3", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
|
||
{"instance_id": "locomo_conv-47_q092", "cell": "no-context", "question": "What did John create for the charitable foundation that helped generate reports for analysis?", "ground_truth": "computer application on smartphones", "context": "Session 11 (5:00 pm on 11 May, 2022):\nJohn: Previously, this foundation used paper records and all inventory was recorded manually. I made an application that structured their work, and now everything they need for inventory is in one application on their smartphone.", "model_answer": "A database", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
|
||
{"instance_id": "locomo_conv-30_q057", "cell": "no-context", "question": "What advice does Gina give to Jon about running a successful business?", "ground_truth": "build relationships with customers, create a strong brand image, stay positive", "context": "Session 7 (7:28 pm on 23 March, 2023):\nJon: Yeah, brand identity is key. Make sure yours stands out. Also be sure to build relationships with your customers – let them know you care. And don't forget to stay positive and motivate others. Your energy will be contagious!", "model_answer": "Focus on the customer.", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-30_q029", "cell": "no-context", "question": "Which cities has Jon visited?", "ground_truth": "Paris, Rome", "context": "Session 2 (2:32 pm on 29 January, 2023):\nJon: Hey Gina! Thanks for asking. I'm on the hunt for the ideal spot for my dance studio and it's been quite a journey! I've been looking at different places and picturing how the space would look. I even found a place with great natural light! Oh, I've been to Paris yesterday! It was sooo cool. [image: a photo of a bathroom with a blue floor and a pink wall]\n\nSession 15 (10:04 am on 19 June, 2023):\nJon: Hey Gina, hope you're doing great! Still working on my biz. Took a short trip last week to Rome to clear my mind a little.", "model_answer": "Unknown.", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
|
||
{"instance_id": "locomo_conv-44_q066", "cell": "no-context", "question": "What type of dog was Andrew looking to adopt based on his living space?", "ground_truth": "smaller dog", "context": "Session 5 (10:47 am on 6 May, 2023):\nAndrew: Meeting all these adorable pups has been awesome! For those considering getting a pup, the size of living space and the exercise needs of the breed are important. For me, a person living in an apartment, a smaller dog would be best, but if one is active, consider getting one that loves to play and run.", "model_answer": "A small dog", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-44_q002", "cell": "no-context", "question": "What kind of indoor activities has Andrew pursued with his girlfriend?", "ground_truth": "boardgames, volunteering at pet shelter, wine tasting, growing flowers", "context": "Session 13 (3:52 pm on 27 July, 2023):\nAndrew: Hey Audrey! How are you? My GF and I just had a great experience volunteering at a pet shelter on Monday - it was so rewarding! We loved spending time with those cute animals and it gave us so much joy. It was so rewarding, it reminded me just how much I love them!\n\nSession 19 (5:53 pm on 24 September, 2023):\nAndrew: Yeah! They really do bring so much into our lives - it's amazing to watch them interact. Here's something I've been taking care of lately. Look at those flowers! [image: a photo of a balcony with a bunch of flowers on it]\n\nSession 23 (4:22 pm on 13 October, 2023):\nAndrew: Hey Audrey, it's been a busy week for me. Last Tuesday, my gf, Toby, and I had a really awesome night playing board games. It was really nice. What's been up with you lately? [image: a photography of a dog sitting at a chess board with a chess set in front of him]\n\nSession 25 (10:14 am on 24 October, 2023):\nAndrew: Hi Audrey! How have you been lately? My girlfriend and I went to this awesome wine tasting last weekend. It was great! We tried so many unique wines and learned a lot. I was surprised at how much I enjoyed it. A reminder to step out of the comfort zone!", "model_answer": "Board games", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-47_q046", "cell": "no-context", "question": "When did John and James meet at McGee's bar?", "ground_truth": "August 27, 2022", "context": "Session 21 (9:18 pm on 26 August, 2022):\nJohn: Great, then I agree! See you tomorrow at McGee's Pub!", "model_answer": "Never", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F3", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F3"}
|
||
{"instance_id": "locomo_conv-43_q076", "cell": "no-context", "question": "What kind of picture did Tim share as part of their Harry Potter book collection?", "ground_truth": "MinaLima's creation from the Harry Potter films", "context": "Session 2 (5:08 pm on 15 June, 2023):\nTim: Thanks! That picture is from MinaLima. They created all the props for the Harry Potter films, and I love their work. It's like having a piece of the wizarding world at home!", "model_answer": "A selfie", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F4", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
|
||
{"instance_id": "locomo_conv-41_q065", "cell": "no-context", "question": "What is John's main focus in local politics?", "ground_truth": "Improving education and infrastructure", "context": "Session 1 (11:01 am on 17 December, 2022):\nJohn: I'm passionate about improving education and infrastructure in our community. Those are my main focuses.", "model_answer": "Unknown which John.", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
|
||
{"instance_id": "locomo_conv-50_q086", "cell": "oracle-context", "question": "What sports activity is Calvin planning to try after the tour with Frank Ocean?", "ground_truth": "Skiing", "context": "Session 9 (3:15 pm on 21 June, 2023):\nCalvin: Nope, not yet! I haven't been there before, but it's on my to-do list for after my tour with Frank Ocean ends. Can't wait to see them in person! Look at that snowy peak. [image: a photo of a person on skis on a snowy mountain]", "model_answer": "Skiing", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-26_q059", "cell": "oracle-context", "question": "Would Caroline be considered religious?", "ground_truth": "Somewhat, but not extremely religious", "context": "Session 12 (1:50 pm on 17 August, 2023):\nCaroline: Hey Mel! How're ya doin'? Recently, I had a not-so-great experience on a hike. I ran into a group of religious conservatives who said something that really upset me. It made me think how much work we still have to do for LGBTQ rights. It's been so helpful to have people around me who accept and support me, so I know I'll be ok!\n\nSession 14 (1:33 pm on 25 August, 2023):\nCaroline: Thanks! It was made for a local church and shows time changing our lives. I made it to show my own journey as a transgender woman and how we should accept growth and change. [image: a photo of a large stained glass window in a church]", "model_answer": "Yes", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-44_q000", "cell": "oracle-context", "question": "Which year did Audrey adopt the first three of her dogs?", "ground_truth": "2020", "context": "Session 1 (1:10 pm on 27 March, 2023):\nAudrey: I've had them for 3 years! Their names are Pepper, Precious and Panda. I can't live without my little ones!", "model_answer": "2020", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-41_q080", "cell": "oracle-context", "question": "What did Maria make for her home to remind her of a trip to England?", "ground_truth": "painting of a castle on a hill", "context": "Session 8 (6:03 pm on 6 March, 2023):\nMaria: Thanks, John! I got the idea from that trip to England a few years ago - I was mesmerized by the castles. I wanted to bring that same magic to my home, like a reminder of the world's beauty. I made a painting too! [image: a photography of a painting of a castle on a hill]", "model_answer": "A painting", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-42_q001", "cell": "oracle-context", "question": "What kind of interests do Joanna and Nate share?", "ground_truth": "Watching movies, making desserts", "context": "Session 1 (7:31 pm on 21 January, 2022):\nJoanna: Yeah! Besides writing, I also enjoy reading, watching movies, and exploring nature. Anything else you enjoy doing, Nate?\nNate: Playing video games and watching movies are my main hobbies.\nJoanna: Cool, Nate! So we both have similar interests. What type of movies do you like best?\n\nSession 3 (9:27 am on 7 February, 2022):\nNate: Thanks, Joanna. Not much has changed for me, but I just discovered that I can make coconut milk icecream and gave it a try. It was actually pretty good, so I'm proud of myself. [image: a photo of a bowl of ice cream with a spoon in it]\n\nSession 4 (1:07 pm on 25 February, 2022):\nNate: Thanks, Joanna! It means a lot that you enjoy the desserts I bake.\n\nSession 10 (11:54 am on 2 May, 2022):\nJoanna: Not much is new other than the screenplay. Been working on some projects and testing out dairy-free dessert recipes for friends and fam. Here's a pic of a cake I made recently! [image: a photo of a cake with white frosting on a wooden table]\n\nSession 20 (6:03 pm on 5 September, 2022):\nJoanna: Hey Nate! Cute turtles! Bummer about the setback. Any positive vibes comin' your way? I just revised on of my old recipes and made this! [image: a photo of a piece of cake with strawberries and chocolate]", "model_answer": "Movies", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-43_q155", "cell": "oracle-context", "question": "How did John feel after being able to jog without pain?", "ground_truth": "It was a huge success.", "context": "Session 24 (3:37 pm on 16 December, 2023):\nJohn: It was great! After being out for so long, jogging without any pain was a huge success. My wife and I hosted a small get-together with friends and family to celebrate.", "model_answer": "Great", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-50_q015", "cell": "oracle-context", "question": "Why does Dave regularly visit parks?", "ground_truth": "because it relaxes and calms him", "context": "Session 1 (11:53 am on 23 March, 2023):\nDave: That's great, can't wait for you to return from your trip. I've been spending lots of time at this beautiful park lately - so calming. Have you been there before? [image: a photography of a park with a lake and a few boats]\n\nSession 8 (2:31 pm on 9 June, 2023):\nDave: Hey, nice photo of the studio! Working in a team yields incredible results, well done! Keep pushing it! I've been doing good - thanks for asking. I've been exploring some parks on the weekends to relax - it's so peaceful being surrounded by nature. Are there any chill spots you enjoy in Boston?", "model_answer": "To relax", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-48_q020", "cell": "oracle-context", "question": "Which games have Jolene and her partner played together?", "ground_truth": "Detroit, Walking Dead, Battlefield 1, It Takes Two, Overcooked 2", "context": "Session 2 (9:49 am on 27 January, 2023):\nJolene: They are very unusual pets! Here's me and my partner gaming last week - it's so fun. We played the game \"Detroit\" on the console. We are both crazy about this activity! [image: a photo of a person laying in bed with a dog watching tv]\nJolene: We are planning to play \"Walking Dead\" next Saturday.\n\nSession 15 (7:37 pm on 9 July, 2023):\nJolene: Yeah, we love playing \"It takes two\" together! It's a fun team-strategy game and it's competitive. Plus, it's a great way for us to bond. Do you have any activities you like doing with people?\n\nSession 19 (12:52 am on 19 August, 2023):\nJolene: Oh, I forgot to mention Overcooked 2 - this is a good co-op game if you're into hilarious and chaotic cooking. My partner and I often play for bets! I once won three large pizzas!\n\nSession 20 (9:11 am on 21 August, 2023):\nJolene: Long time no talk! We were given a new game for the console last week, it is Battlefield 1. What's been up with you?", "model_answer": "Detroit, It Takes Two, Overcooked 2", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-48_q055", "cell": "oracle-context", "question": "Which pet did Jolene adopt more recently - Susie or Seraphim?", "ground_truth": "Seraphim", "context": "Session 2 (9:49 am on 27 January, 2023):\nJolene: I bought it a year ago in Paris.\nJolene: Even as a child I learned to play on my own.\n\nSession 16 (9:26 am on 1 August, 2023):\nJolene: I adopted her two years ago when I was feeling lonely and wanted some company.", "model_answer": "Susie", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F3", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F3"}
|
||
{"instance_id": "locomo_conv-26_q128", "cell": "oracle-context", "question": "What did Caroline find in her neighborhood during her walk?", "ground_truth": "a rainbow sidewalk", "context": "Session 14 (1:33 pm on 25 August, 2023):\nCaroline: I was out walking in my neighborhood when I came across this cool rainbow sidewalk for Pride Month. It was so vibrant and welcoming, I had to take a picture! It reminds us that love and acceptance are everywhere—even where we least expect it.", "model_answer": "A rainbow sidewalk", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-50_q010", "cell": "oracle-context", "question": "When did Calvin's place get flooded in Tokyo?", "ground_truth": "On a week before 16 May, 2023", "context": "Session 6 (11:50 am on 16 May, 2023):\nCalvin: Hey Dave, not everything has been going smoothly. I had an incident last week where my place got flooded, but thankfully, I managed to save my music gear and favorite microphone. It's been tough, but I'm staying positive and looking forward to getting everything fixed up.", "model_answer": "Last week", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-50_q078", "cell": "oracle-context", "question": "What was the necklace Calvin received meant to remind him of?", "ground_truth": "why he keeps hustling as a musician", "context": "Session 4 (6:24 pm on 1 May, 2023):\nCalvin: Thanks, Dave! I got it from another artist as a gift - it's a great reminder of why I keep hustling as a musician!", "model_answer": "Why he keeps hustling as a musician", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-47_q092", "cell": "oracle-context", "question": "What did John create for the charitable foundation that helped generate reports for analysis?", "ground_truth": "computer application on smartphones", "context": "Session 11 (5:00 pm on 11 May, 2022):\nJohn: Previously, this foundation used paper records and all inventory was recorded manually. I made an application that structured their work, and now everything they need for inventory is in one application on their smartphone.", "model_answer": "An application", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-30_q057", "cell": "oracle-context", "question": "What advice does Gina give to Jon about running a successful business?", "ground_truth": "build relationships with customers, create a strong brand image, stay positive", "context": "Session 7 (7:28 pm on 23 March, 2023):\nJon: Yeah, brand identity is key. Make sure yours stands out. Also be sure to build relationships with your customers – let them know you care. And don't forget to stay positive and motivate others. Your energy will be contagious!", "model_answer": "Not mentioned.", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
|
||
{"instance_id": "locomo_conv-30_q029", "cell": "oracle-context", "question": "Which cities has Jon visited?", "ground_truth": "Paris, Rome", "context": "Session 2 (2:32 pm on 29 January, 2023):\nJon: Hey Gina! Thanks for asking. I'm on the hunt for the ideal spot for my dance studio and it's been quite a journey! I've been looking at different places and picturing how the space would look. I even found a place with great natural light! Oh, I've been to Paris yesterday! It was sooo cool. [image: a photo of a bathroom with a blue floor and a pink wall]\n\nSession 15 (10:04 am on 19 June, 2023):\nJon: Hey Gina, hope you're doing great! Still working on my biz. Took a short trip last week to Rome to clear my mind a little.", "model_answer": "Paris and Rome", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-44_q066", "cell": "oracle-context", "question": "What type of dog was Andrew looking to adopt based on his living space?", "ground_truth": "smaller dog", "context": "Session 5 (10:47 am on 6 May, 2023):\nAndrew: Meeting all these adorable pups has been awesome! For those considering getting a pup, the size of living space and the exercise needs of the breed are important. For me, a person living in an apartment, a smaller dog would be best, but if one is active, consider getting one that loves to play and run.", "model_answer": "A smaller dog", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-44_q002", "cell": "oracle-context", "question": "What kind of indoor activities has Andrew pursued with his girlfriend?", "ground_truth": "boardgames, volunteering at pet shelter, wine tasting, growing flowers", "context": "Session 13 (3:52 pm on 27 July, 2023):\nAndrew: Hey Audrey! How are you? My GF and I just had a great experience volunteering at a pet shelter on Monday - it was so rewarding! We loved spending time with those cute animals and it gave us so much joy. It was so rewarding, it reminded me just how much I love them!\n\nSession 19 (5:53 pm on 24 September, 2023):\nAndrew: Yeah! They really do bring so much into our lives - it's amazing to watch them interact. Here's something I've been taking care of lately. Look at those flowers! [image: a photo of a balcony with a bunch of flowers on it]\n\nSession 23 (4:22 pm on 13 October, 2023):\nAndrew: Hey Audrey, it's been a busy week for me. Last Tuesday, my gf, Toby, and I had a really awesome night playing board games. It was really nice. What's been up with you lately? [image: a photography of a dog sitting at a chess board with a chess set in front of him]\n\nSession 25 (10:14 am on 24 October, 2023):\nAndrew: Hi Audrey! How have you been lately? My girlfriend and I went to this awesome wine tasting last weekend. It was great! We tried so many unique wines and learned a lot. I was surprised at how much I enjoyed it. A reminder to step out of the comfort zone!", "model_answer": "Board games.", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-47_q046", "cell": "oracle-context", "question": "When did John and James meet at McGee's bar?", "ground_truth": "August 27, 2022", "context": "Session 21 (9:18 pm on 26 August, 2022):\nJohn: Great, then I agree! See you tomorrow at McGee's Pub!", "model_answer": "27 August, 2022", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-43_q076", "cell": "oracle-context", "question": "What kind of picture did Tim share as part of their Harry Potter book collection?", "ground_truth": "MinaLima's creation from the Harry Potter films", "context": "Session 2 (5:08 pm on 15 June, 2023):\nTim: Thanks! That picture is from MinaLima. They created all the props for the Harry Potter films, and I love their work. It's like having a piece of the wizarding world at home!", "model_answer": "MinaLima", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-41_q065", "cell": "oracle-context", "question": "What is John's main focus in local politics?", "ground_truth": "Improving education and infrastructure", "context": "Session 1 (11:01 am on 17 December, 2022):\nJohn: I'm passionate about improving education and infrastructure in our community. Those are my main focuses.", "model_answer": "Education and infrastructure", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-50_q086", "cell": "full-context", "question": "What sports activity is Calvin planning to try after the tour with Frank Ocean?", "ground_truth": "Skiing", "context": "Session 9 (3:15 pm on 21 June, 2023):\nCalvin: Nope, not yet! I haven't been there before, but it's on my to-do list for after my tour with Frank Ocean ends. Can't wait to see them in person! Look at that snowy peak. [image: a photo of a person on skis on a snowy mountain]", "model_answer": "skiing", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-26_q059", "cell": "full-context", "question": "Would Caroline be considered religious?", "ground_truth": "Somewhat, but not extremely religious", "context": "Session 12 (1:50 pm on 17 August, 2023):\nCaroline: Hey Mel! How're ya doin'? Recently, I had a not-so-great experience on a hike. I ran into a group of religious conservatives who said something that really upset me. It made me think how much work we still have to do for LGBTQ rights. It's been so helpful to have people around me who accept and support me, so I know I'll be ok!\n\nSession 14 (1:33 pm on 25 August, 2023):\nCaroline: Thanks! It was made for a local church and shows time changing our lives. I made it to show my own journey as a transgender woman and how we should accept growth and change. [image: a photo of a large stained glass window in a church]", "model_answer": "unknown", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
|
||
{"instance_id": "locomo_conv-44_q000", "cell": "full-context", "question": "Which year did Audrey adopt the first three of her dogs?", "ground_truth": "2020", "context": "Session 1 (1:10 pm on 27 March, 2023):\nAudrey: I've had them for 3 years! Their names are Pepper, Precious and Panda. I can't live without my little ones!", "model_answer": "2020", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-41_q080", "cell": "full-context", "question": "What did Maria make for her home to remind her of a trip to England?", "ground_truth": "painting of a castle on a hill", "context": "Session 8 (6:03 pm on 6 March, 2023):\nMaria: Thanks, John! I got the idea from that trip to England a few years ago - I was mesmerized by the castles. I wanted to bring that same magic to my home, like a reminder of the world's beauty. I made a painting too! [image: a photography of a painting of a castle on a hill]", "model_answer": "a painting", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-42_q001", "cell": "full-context", "question": "What kind of interests do Joanna and Nate share?", "ground_truth": "Watching movies, making desserts", "context": "Session 1 (7:31 pm on 21 January, 2022):\nJoanna: Yeah! Besides writing, I also enjoy reading, watching movies, and exploring nature. Anything else you enjoy doing, Nate?\nNate: Playing video games and watching movies are my main hobbies.\nJoanna: Cool, Nate! So we both have similar interests. What type of movies do you like best?\n\nSession 3 (9:27 am on 7 February, 2022):\nNate: Thanks, Joanna. Not much has changed for me, but I just discovered that I can make coconut milk icecream and gave it a try. It was actually pretty good, so I'm proud of myself. [image: a photo of a bowl of ice cream with a spoon in it]\n\nSession 4 (1:07 pm on 25 February, 2022):\nNate: Thanks, Joanna! It means a lot that you enjoy the desserts I bake.\n\nSession 10 (11:54 am on 2 May, 2022):\nJoanna: Not much is new other than the screenplay. Been working on some projects and testing out dairy-free dessert recipes for friends and fam. Here's a pic of a cake I made recently! [image: a photo of a cake with white frosting on a wooden table]\n\nSession 20 (6:03 pm on 5 September, 2022):\nJoanna: Hey Nate! Cute turtles! Bummer about the setback. Any positive vibes comin' your way? I just revised on of my old recipes and made this! [image: a photo of a piece of cake with strawberries and chocolate]", "model_answer": "watching movies", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-43_q155", "cell": "full-context", "question": "How did John feel after being able to jog without pain?", "ground_truth": "It was a huge success.", "context": "Session 24 (3:37 pm on 16 December, 2023):\nJohn: It was great! After being out for so long, jogging without any pain was a huge success. My wife and I hosted a small get-together with friends and family to celebrate.", "model_answer": "great", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-50_q015", "cell": "full-context", "question": "Why does Dave regularly visit parks?", "ground_truth": "because it relaxes and calms him", "context": "Session 1 (11:53 am on 23 March, 2023):\nDave: That's great, can't wait for you to return from your trip. I've been spending lots of time at this beautiful park lately - so calming. Have you been there before? [image: a photography of a park with a lake and a few boats]\n\nSession 8 (2:31 pm on 9 June, 2023):\nDave: Hey, nice photo of the studio! Working in a team yields incredible results, well done! Keep pushing it! I've been doing good - thanks for asking. I've been exploring some parks on the weekends to relax - it's so peaceful being surrounded by nature. Are there any chill spots you enjoy in Boston?", "model_answer": "to relax", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-48_q020", "cell": "full-context", "question": "Which games have Jolene and her partner played together?", "ground_truth": "Detroit, Walking Dead, Battlefield 1, It Takes Two, Overcooked 2", "context": "Session 2 (9:49 am on 27 January, 2023):\nJolene: They are very unusual pets! Here's me and my partner gaming last week - it's so fun. We played the game \"Detroit\" on the console. We are both crazy about this activity! [image: a photo of a person laying in bed with a dog watching tv]\nJolene: We are planning to play \"Walking Dead\" next Saturday.\n\nSession 15 (7:37 pm on 9 July, 2023):\nJolene: Yeah, we love playing \"It takes two\" together! It's a fun team-strategy game and it's competitive. Plus, it's a great way for us to bond. Do you have any activities you like doing with people?\n\nSession 19 (12:52 am on 19 August, 2023):\nJolene: Oh, I forgot to mention Overcooked 2 - this is a good co-op game if you're into hilarious and chaotic cooking. My partner and I often play for bets! I once won three large pizzas!\n\nSession 20 (9:11 am on 21 August, 2023):\nJolene: Long time no talk! We were given a new game for the console last week, it is Battlefield 1. What's been up with you?", "model_answer": "Detroit, It takes two, Overcooked 2", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-48_q055", "cell": "full-context", "question": "Which pet did Jolene adopt more recently - Susie or Seraphim?", "ground_truth": "Seraphim", "context": "Session 2 (9:49 am on 27 January, 2023):\nJolene: I bought it a year ago in Paris.\nJolene: Even as a child I learned to play on my own.\n\nSession 16 (9:26 am on 1 August, 2023):\nJolene: I adopted her two years ago when I was feeling lonely and wanted some company.", "model_answer": "unknown", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
|
||
{"instance_id": "locomo_conv-26_q128", "cell": "full-context", "question": "What did Caroline find in her neighborhood during her walk?", "ground_truth": "a rainbow sidewalk", "context": "Session 14 (1:33 pm on 25 August, 2023):\nCaroline: I was out walking in my neighborhood when I came across this cool rainbow sidewalk for Pride Month. It was so vibrant and welcoming, I had to take a picture! It reminds us that love and acceptance are everywhere—even where we least expect it.", "model_answer": "rainbow sidewalk", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-50_q010", "cell": "full-context", "question": "When did Calvin's place get flooded in Tokyo?", "ground_truth": "On a week before 16 May, 2023", "context": "Session 6 (11:50 am on 16 May, 2023):\nCalvin: Hey Dave, not everything has been going smoothly. I had an incident last week where my place got flooded, but thankfully, I managed to save my music gear and favorite microphone. It's been tough, but I'm staying positive and looking forward to getting everything fixed up.", "model_answer": "last week", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-50_q078", "cell": "full-context", "question": "What was the necklace Calvin received meant to remind him of?", "ground_truth": "why he keeps hustling as a musician", "context": "Session 4 (6:24 pm on 1 May, 2023):\nCalvin: Thanks, Dave! I got it from another artist as a gift - it's a great reminder of why I keep hustling as a musician!", "model_answer": "why I keep hustling as a musician", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-47_q092", "cell": "full-context", "question": "What did John create for the charitable foundation that helped generate reports for analysis?", "ground_truth": "computer application on smartphones", "context": "Session 11 (5:00 pm on 11 May, 2022):\nJohn: Previously, this foundation used paper records and all inventory was recorded manually. I made an application that structured their work, and now everything they need for inventory is in one application on their smartphone.", "model_answer": "an application", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-30_q057", "cell": "full-context", "question": "What advice does Gina give to Jon about running a successful business?", "ground_truth": "build relationships with customers, create a strong brand image, stay positive", "context": "Session 7 (7:28 pm on 23 March, 2023):\nJon: Yeah, brand identity is key. Make sure yours stands out. Also be sure to build relationships with your customers – let them know you care. And don't forget to stay positive and motivate others. Your energy will be contagious!", "model_answer": "unknown", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
|
||
{"instance_id": "locomo_conv-30_q029", "cell": "full-context", "question": "Which cities has Jon visited?", "ground_truth": "Paris, Rome", "context": "Session 2 (2:32 pm on 29 January, 2023):\nJon: Hey Gina! Thanks for asking. I'm on the hunt for the ideal spot for my dance studio and it's been quite a journey! I've been looking at different places and picturing how the space would look. I even found a place with great natural light! Oh, I've been to Paris yesterday! It was sooo cool. [image: a photo of a bathroom with a blue floor and a pink wall]\n\nSession 15 (10:04 am on 19 June, 2023):\nJon: Hey Gina, hope you're doing great! Still working on my biz. Took a short trip last week to Rome to clear my mind a little.", "model_answer": "Paris Rome", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-44_q066", "cell": "full-context", "question": "What type of dog was Andrew looking to adopt based on his living space?", "ground_truth": "smaller dog", "context": "Session 5 (10:47 am on 6 May, 2023):\nAndrew: Meeting all these adorable pups has been awesome! For those considering getting a pup, the size of living space and the exercise needs of the breed are important. For me, a person living in an apartment, a smaller dog would be best, but if one is active, consider getting one that loves to play and run.", "model_answer": "smaller dog", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-44_q002", "cell": "full-context", "question": "What kind of indoor activities has Andrew pursued with his girlfriend?", "ground_truth": "boardgames, volunteering at pet shelter, wine tasting, growing flowers", "context": "Session 13 (3:52 pm on 27 July, 2023):\nAndrew: Hey Audrey! How are you? My GF and I just had a great experience volunteering at a pet shelter on Monday - it was so rewarding! We loved spending time with those cute animals and it gave us so much joy. It was so rewarding, it reminded me just how much I love them!\n\nSession 19 (5:53 pm on 24 September, 2023):\nAndrew: Yeah! They really do bring so much into our lives - it's amazing to watch them interact. Here's something I've been taking care of lately. Look at those flowers! [image: a photo of a balcony with a bunch of flowers on it]\n\nSession 23 (4:22 pm on 13 October, 2023):\nAndrew: Hey Audrey, it's been a busy week for me. Last Tuesday, my gf, Toby, and I had a really awesome night playing board games. It was really nice. What's been up with you lately? [image: a photography of a dog sitting at a chess board with a chess set in front of him]\n\nSession 25 (10:14 am on 24 October, 2023):\nAndrew: Hi Audrey! How have you been lately? My girlfriend and I went to this awesome wine tasting last weekend. It was great! We tried so many unique wines and learned a lot. I was surprised at how much I enjoyed it. A reminder to step out of the comfort zone!", "model_answer": "playing board games", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-47_q046", "cell": "full-context", "question": "When did John and James meet at McGee's bar?", "ground_truth": "August 27, 2022", "context": "Session 21 (9:18 pm on 26 August, 2022):\nJohn: Great, then I agree! See you tomorrow at McGee's Pub!", "model_answer": "tomorrow", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F3"}
|
||
{"instance_id": "locomo_conv-43_q076", "cell": "full-context", "question": "What kind of picture did Tim share as part of their Harry Potter book collection?", "ground_truth": "MinaLima's creation from the Harry Potter films", "context": "Session 2 (5:08 pm on 15 June, 2023):\nTim: Thanks! That picture is from MinaLima. They created all the props for the Harry Potter films, and I love their work. It's like having a piece of the wizarding world at home!", "model_answer": "from MinaLima", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-41_q065", "cell": "full-context", "question": "What is John's main focus in local politics?", "ground_truth": "Improving education and infrastructure", "context": "Session 1 (11:01 am on 17 December, 2022):\nJohn: I'm passionate about improving education and infrastructure in our community. Those are my main focuses.", "model_answer": "improving education and infrastructure", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-50_q086", "cell": "retrieval", "question": "What sports activity is Calvin planning to try after the tour with Frank Ocean?", "ground_truth": "Skiing", "context": "Session 9 (3:15 pm on 21 June, 2023):\nCalvin: Nope, not yet! I haven't been there before, but it's on my to-do list for after my tour with Frank Ocean ends. Can't wait to see them in person! Look at that snowy peak. [image: a photo of a person on skis on a snowy mountain]", "model_answer": "Not mentioned", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
|
||
{"instance_id": "locomo_conv-26_q059", "cell": "retrieval", "question": "Would Caroline be considered religious?", "ground_truth": "Somewhat, but not extremely religious", "context": "Session 12 (1:50 pm on 17 August, 2023):\nCaroline: Hey Mel! How're ya doin'? Recently, I had a not-so-great experience on a hike. I ran into a group of religious conservatives who said something that really upset me. It made me think how much work we still have to do for LGBTQ rights. It's been so helpful to have people around me who accept and support me, so I know I'll be ok!\n\nSession 14 (1:33 pm on 25 August, 2023):\nCaroline: Thanks! It was made for a local church and shows time changing our lives. I made it to show my own journey as a transgender woman and how we should accept growth and change. [image: a photo of a large stained glass window in a church]", "model_answer": "No.", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F3", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F3"}
|
||
{"instance_id": "locomo_conv-44_q000", "cell": "retrieval", "question": "Which year did Audrey adopt the first three of her dogs?", "ground_truth": "2020", "context": "Session 1 (1:10 pm on 27 March, 2023):\nAudrey: I've had them for 3 years! Their names are Pepper, Precious and Panda. I can't live without my little ones!", "model_answer": "Not mentioned.", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
|
||
{"instance_id": "locomo_conv-41_q080", "cell": "retrieval", "question": "What did Maria make for her home to remind her of a trip to England?", "ground_truth": "painting of a castle on a hill", "context": "Session 8 (6:03 pm on 6 March, 2023):\nMaria: Thanks, John! I got the idea from that trip to England a few years ago - I was mesmerized by the castles. I wanted to bring that same magic to my home, like a reminder of the world's beauty. I made a painting too! [image: a photography of a painting of a castle on a hill]", "model_answer": "A painting.", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-42_q001", "cell": "retrieval", "question": "What kind of interests do Joanna and Nate share?", "ground_truth": "Watching movies, making desserts", "context": "Session 1 (7:31 pm on 21 January, 2022):\nJoanna: Yeah! Besides writing, I also enjoy reading, watching movies, and exploring nature. Anything else you enjoy doing, Nate?\nNate: Playing video games and watching movies are my main hobbies.\nJoanna: Cool, Nate! So we both have similar interests. What type of movies do you like best?\n\nSession 3 (9:27 am on 7 February, 2022):\nNate: Thanks, Joanna. Not much has changed for me, but I just discovered that I can make coconut milk icecream and gave it a try. It was actually pretty good, so I'm proud of myself. [image: a photo of a bowl of ice cream with a spoon in it]\n\nSession 4 (1:07 pm on 25 February, 2022):\nNate: Thanks, Joanna! It means a lot that you enjoy the desserts I bake.\n\nSession 10 (11:54 am on 2 May, 2022):\nJoanna: Not much is new other than the screenplay. Been working on some projects and testing out dairy-free dessert recipes for friends and fam. Here's a pic of a cake I made recently! [image: a photo of a cake with white frosting on a wooden table]\n\nSession 20 (6:03 pm on 5 September, 2022):\nJoanna: Hey Nate! Cute turtles! Bummer about the setback. Any positive vibes comin' your way? I just revised on of my old recipes and made this! [image: a photo of a piece of cake with strawberries and chocolate]", "model_answer": "Movies and nature.", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F3", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F3"}
|
||
{"instance_id": "locomo_conv-43_q155", "cell": "retrieval", "question": "How did John feel after being able to jog without pain?", "ground_truth": "It was a huge success.", "context": "Session 24 (3:37 pm on 16 December, 2023):\nJohn: It was great! After being out for so long, jogging without any pain was a huge success. My wife and I hosted a small get-together with friends and family to celebrate.", "model_answer": "Relieved", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F3", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
|
||
{"instance_id": "locomo_conv-50_q015", "cell": "retrieval", "question": "Why does Dave regularly visit parks?", "ground_truth": "because it relaxes and calms him", "context": "Session 1 (11:53 am on 23 March, 2023):\nDave: That's great, can't wait for you to return from your trip. I've been spending lots of time at this beautiful park lately - so calming. Have you been there before? [image: a photography of a park with a lake and a few boats]\n\nSession 8 (2:31 pm on 9 June, 2023):\nDave: Hey, nice photo of the studio! Working in a team yields incredible results, well done! Keep pushing it! I've been doing good - thanks for asking. I've been exploring some parks on the weekends to relax - it's so peaceful being surrounded by nature. Are there any chill spots you enjoy in Boston?", "model_answer": "To relax, recharge, and hang out with friends.", "opus_verdict": "incorrect", "opus_failure_mode": "F4", "gpt_verdict": "incorrect", "gpt_failure_mode": "F4", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
|
||
{"instance_id": "locomo_conv-48_q020", "cell": "retrieval", "question": "Which games have Jolene and her partner played together?", "ground_truth": "Detroit, Walking Dead, Battlefield 1, It Takes Two, Overcooked 2", "context": "Session 2 (9:49 am on 27 January, 2023):\nJolene: They are very unusual pets! Here's me and my partner gaming last week - it's so fun. We played the game \"Detroit\" on the console. We are both crazy about this activity! [image: a photo of a person laying in bed with a dog watching tv]\nJolene: We are planning to play \"Walking Dead\" next Saturday.\n\nSession 15 (7:37 pm on 9 July, 2023):\nJolene: Yeah, we love playing \"It takes two\" together! It's a fun team-strategy game and it's competitive. Plus, it's a great way for us to bond. Do you have any activities you like doing with people?\n\nSession 19 (12:52 am on 19 August, 2023):\nJolene: Oh, I forgot to mention Overcooked 2 - this is a good co-op game if you're into hilarious and chaotic cooking. My partner and I often play for bets! I once won three large pizzas!\n\nSession 20 (9:11 am on 21 August, 2023):\nJolene: Long time no talk! We were given a new game for the console last week, it is Battlefield 1. What's been up with you?", "model_answer": "Detroit, Overcooked 2, It Takes Two", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-48_q055", "cell": "retrieval", "question": "Which pet did Jolene adopt more recently - Susie or Seraphim?", "ground_truth": "Seraphim", "context": "Session 2 (9:49 am on 27 January, 2023):\nJolene: I bought it a year ago in Paris.\nJolene: Even as a child I learned to play on my own.\n\nSession 16 (9:26 am on 1 August, 2023):\nJolene: I adopted her two years ago when I was feeling lonely and wanted some company.", "model_answer": "Seraphim", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-26_q128", "cell": "retrieval", "question": "What did Caroline find in her neighborhood during her walk?", "ground_truth": "a rainbow sidewalk", "context": "Session 14 (1:33 pm on 25 August, 2023):\nCaroline: I was out walking in my neighborhood when I came across this cool rainbow sidewalk for Pride Month. It was so vibrant and welcoming, I had to take a picture! It reminds us that love and acceptance are everywhere—even where we least expect it.", "model_answer": "A rainbow sidewalk.", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-50_q010", "cell": "retrieval", "question": "When did Calvin's place get flooded in Tokyo?", "ground_truth": "On a week before 16 May, 2023", "context": "Session 6 (11:50 am on 16 May, 2023):\nCalvin: Hey Dave, not everything has been going smoothly. I had an incident last week where my place got flooded, but thankfully, I managed to save my music gear and favorite microphone. It's been tough, but I'm staying positive and looking forward to getting everything fixed up.", "model_answer": "Last week.", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-50_q078", "cell": "retrieval", "question": "What was the necklace Calvin received meant to remind him of?", "ground_truth": "why he keeps hustling as a musician", "context": "Session 4 (6:24 pm on 1 May, 2023):\nCalvin: Thanks, Dave! I got it from another artist as a gift - it's a great reminder of why I keep hustling as a musician!", "model_answer": "Why he keeps hustling as a musician.", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-47_q092", "cell": "retrieval", "question": "What did John create for the charitable foundation that helped generate reports for analysis?", "ground_truth": "computer application on smartphones", "context": "Session 11 (5:00 pm on 11 May, 2022):\nJohn: Previously, this foundation used paper records and all inventory was recorded manually. I made an application that structured their work, and now everything they need for inventory is in one application on their smartphone.", "model_answer": "A software tool", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-30_q057", "cell": "retrieval", "question": "What advice does Gina give to Jon about running a successful business?", "ground_truth": "build relationships with customers, create a strong brand image, stay positive", "context": "Session 7 (7:28 pm on 23 March, 2023):\nJon: Yeah, brand identity is key. Make sure yours stands out. Also be sure to build relationships with your customers – let them know you care. And don't forget to stay positive and motivate others. Your energy will be contagious!", "model_answer": "Stay passionate, focused, and resilient; believe in yourself, keep going, and stay open to learning.", "opus_verdict": "incorrect", "opus_failure_mode": "F4", "gpt_verdict": "incorrect", "gpt_failure_mode": "F4", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
|
||
{"instance_id": "locomo_conv-30_q029", "cell": "retrieval", "question": "Which cities has Jon visited?", "ground_truth": "Paris, Rome", "context": "Session 2 (2:32 pm on 29 January, 2023):\nJon: Hey Gina! Thanks for asking. I'm on the hunt for the ideal spot for my dance studio and it's been quite a journey! I've been looking at different places and picturing how the space would look. I even found a place with great natural light! Oh, I've been to Paris yesterday! It was sooo cool. [image: a photo of a bathroom with a blue floor and a pink wall]\n\nSession 15 (10:04 am on 19 June, 2023):\nJon: Hey Gina, hope you're doing great! Still working on my biz. Took a short trip last week to Rome to clear my mind a little.", "model_answer": "Paris", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-44_q066", "cell": "retrieval", "question": "What type of dog was Andrew looking to adopt based on his living space?", "ground_truth": "smaller dog", "context": "Session 5 (10:47 am on 6 May, 2023):\nAndrew: Meeting all these adorable pups has been awesome! For those considering getting a pup, the size of living space and the exercise needs of the breed are important. For me, a person living in an apartment, a smaller dog would be best, but if one is active, consider getting one that loves to play and run.", "model_answer": "A smaller dog.", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-44_q002", "cell": "retrieval", "question": "What kind of indoor activities has Andrew pursued with his girlfriend?", "ground_truth": "boardgames, volunteering at pet shelter, wine tasting, growing flowers", "context": "Session 13 (3:52 pm on 27 July, 2023):\nAndrew: Hey Audrey! How are you? My GF and I just had a great experience volunteering at a pet shelter on Monday - it was so rewarding! We loved spending time with those cute animals and it gave us so much joy. It was so rewarding, it reminded me just how much I love them!\n\nSession 19 (5:53 pm on 24 September, 2023):\nAndrew: Yeah! They really do bring so much into our lives - it's amazing to watch them interact. Here's something I've been taking care of lately. Look at those flowers! [image: a photo of a balcony with a bunch of flowers on it]\n\nSession 23 (4:22 pm on 13 October, 2023):\nAndrew: Hey Audrey, it's been a busy week for me. Last Tuesday, my gf, Toby, and I had a really awesome night playing board games. It was really nice. What's been up with you lately? [image: a photography of a dog sitting at a chess board with a chess set in front of him]\n\nSession 25 (10:14 am on 24 October, 2023):\nAndrew: Hi Audrey! How have you been lately? My girlfriend and I went to this awesome wine tasting last weekend. It was great! We tried so many unique wines and learned a lot. I was surprised at how much I enjoyed it. A reminder to step out of the comfort zone!", "model_answer": "Wine tasting", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-47_q046", "cell": "retrieval", "question": "When did John and James meet at McGee's bar?", "ground_truth": "August 27, 2022", "context": "Session 21 (9:18 pm on 26 August, 2022):\nJohn: Great, then I agree! See you tomorrow at McGee's Pub!", "model_answer": "Tomorrow", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F3"}
|
||
{"instance_id": "locomo_conv-43_q076", "cell": "retrieval", "question": "What kind of picture did Tim share as part of their Harry Potter book collection?", "ground_truth": "MinaLima's creation from the Harry Potter films", "context": "Session 2 (5:08 pm on 15 June, 2023):\nTim: Thanks! That picture is from MinaLima. They created all the props for the Harry Potter films, and I love their work. It's like having a piece of the wizarding world at home!", "model_answer": "MinaLima props", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-41_q065", "cell": "retrieval", "question": "What is John's main focus in local politics?", "ground_truth": "Improving education and infrastructure", "context": "Session 1 (11:01 am on 17 December, 2022):\nJohn: I'm passionate about improving education and infrastructure in our community. Those are my main focuses.", "model_answer": "Education and infrastructure", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-50_q086", "cell": "agentic", "question": "What sports activity is Calvin planning to try after the tour with Frank Ocean?", "ground_truth": "Skiing", "context": "Session 9 (3:15 pm on 21 June, 2023):\nCalvin: Nope, not yet! I haven't been there before, but it's on my to-do list for after my tour with Frank Ocean ends. Can't wait to see them in person! Look at that snowy peak. [image: a photo of a person on skis on a snowy mountain]", "model_answer": "Max tool turns reached (3 turns, 3 tools used).", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
|
||
{"instance_id": "locomo_conv-26_q059", "cell": "agentic", "question": "Would Caroline be considered religious?", "ground_truth": "Somewhat, but not extremely religious", "context": "Session 12 (1:50 pm on 17 August, 2023):\nCaroline: Hey Mel! How're ya doin'? Recently, I had a not-so-great experience on a hike. I ran into a group of religious conservatives who said something that really upset me. It made me think how much work we still have to do for LGBTQ rights. It's been so helpful to have people around me who accept and support me, so I know I'll be ok!\n\nSession 14 (1:33 pm on 25 August, 2023):\nCaroline: Thanks! It was made for a local church and shows time changing our lives. I made it to show my own journey as a transgender woman and how we should accept growth and change. [image: a photo of a large stained glass window in a church]", "model_answer": "Yes, Caroline has a connection to faith through her heritage (a necklace from Sweden representing \"love, faith and strength\") and by having made a painting for a local church.", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F4", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
|
||
{"instance_id": "locomo_conv-44_q000", "cell": "agentic", "question": "Which year did Audrey adopt the first three of her dogs?", "ground_truth": "2020", "context": "Session 1 (1:10 pm on 27 March, 2023):\nAudrey: I've had them for 3 years! Their names are Pepper, Precious and Panda. I can't live without my little ones!", "model_answer": "Max tool turns reached (3 turns, 3 tools used).", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
|
||
{"instance_id": "locomo_conv-41_q080", "cell": "agentic", "question": "What did Maria make for her home to remind her of a trip to England?", "ground_truth": "painting of a castle on a hill", "context": "Session 8 (6:03 pm on 6 March, 2023):\nMaria: Thanks, John! I got the idea from that trip to England a few years ago - I was mesmerized by the castles. I wanted to bring that same magic to my home, like a reminder of the world's beauty. I made a painting too! [image: a photography of a painting of a castle on a hill]", "model_answer": "a painting", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-42_q001", "cell": "agentic", "question": "What kind of interests do Joanna and Nate share?", "ground_truth": "Watching movies, making desserts", "context": "Session 1 (7:31 pm on 21 January, 2022):\nJoanna: Yeah! Besides writing, I also enjoy reading, watching movies, and exploring nature. Anything else you enjoy doing, Nate?\nNate: Playing video games and watching movies are my main hobbies.\nJoanna: Cool, Nate! So we both have similar interests. What type of movies do you like best?\n\nSession 3 (9:27 am on 7 February, 2022):\nNate: Thanks, Joanna. Not much has changed for me, but I just discovered that I can make coconut milk icecream and gave it a try. It was actually pretty good, so I'm proud of myself. [image: a photo of a bowl of ice cream with a spoon in it]\n\nSession 4 (1:07 pm on 25 February, 2022):\nNate: Thanks, Joanna! It means a lot that you enjoy the desserts I bake.\n\nSession 10 (11:54 am on 2 May, 2022):\nJoanna: Not much is new other than the screenplay. Been working on some projects and testing out dairy-free dessert recipes for friends and fam. Here's a pic of a cake I made recently! [image: a photo of a cake with white frosting on a wooden table]\n\nSession 20 (6:03 pm on 5 September, 2022):\nJoanna: Hey Nate! Cute turtles! Bummer about the setback. Any positive vibes comin' your way? I just revised on of my old recipes and made this! [image: a photo of a piece of cake with strawberries and chocolate]", "model_answer": "movies", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-43_q155", "cell": "agentic", "question": "How did John feel after being able to jog without pain?", "ground_truth": "It was a huge success.", "context": "Session 24 (3:37 pm on 16 December, 2023):\nJohn: It was great! After being out for so long, jogging without any pain was a huge success. My wife and I hosted a small get-together with friends and family to celebrate.", "model_answer": "great", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-50_q015", "cell": "agentic", "question": "Why does Dave regularly visit parks?", "ground_truth": "because it relaxes and calms him", "context": "Session 1 (11:53 am on 23 March, 2023):\nDave: That's great, can't wait for you to return from your trip. I've been spending lots of time at this beautiful park lately - so calming. Have you been there before? [image: a photography of a park with a lake and a few boats]\n\nSession 8 (2:31 pm on 9 June, 2023):\nDave: Hey, nice photo of the studio! Working in a team yields incredible results, well done! Keep pushing it! I've been doing good - thanks for asking. I've been exploring some parks on the weekends to relax - it's so peaceful being surrounded by nature. Are there any chill spots you enjoy in Boston?", "model_answer": "to relax and recharge", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-48_q020", "cell": "agentic", "question": "Which games have Jolene and her partner played together?", "ground_truth": "Detroit, Walking Dead, Battlefield 1, It Takes Two, Overcooked 2", "context": "Session 2 (9:49 am on 27 January, 2023):\nJolene: They are very unusual pets! Here's me and my partner gaming last week - it's so fun. We played the game \"Detroit\" on the console. We are both crazy about this activity! [image: a photo of a person laying in bed with a dog watching tv]\nJolene: We are planning to play \"Walking Dead\" next Saturday.\n\nSession 15 (7:37 pm on 9 July, 2023):\nJolene: Yeah, we love playing \"It takes two\" together! It's a fun team-strategy game and it's competitive. Plus, it's a great way for us to bond. Do you have any activities you like doing with people?\n\nSession 19 (12:52 am on 19 August, 2023):\nJolene: Oh, I forgot to mention Overcooked 2 - this is a good co-op game if you're into hilarious and chaotic cooking. My partner and I often play for bets! I once won three large pizzas!\n\nSession 20 (9:11 am on 21 August, 2023):\nJolene: Long time no talk! We were given a new game for the console last week, it is Battlefield 1. What's been up with you?", "model_answer": "Detroit, Overcooked 2, It Takes Two", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-48_q055", "cell": "agentic", "question": "Which pet did Jolene adopt more recently - Susie or Seraphim?", "ground_truth": "Seraphim", "context": "Session 2 (9:49 am on 27 January, 2023):\nJolene: I bought it a year ago in Paris.\nJolene: Even as a child I learned to play on my own.\n\nSession 16 (9:26 am on 1 August, 2023):\nJolene: I adopted her two years ago when I was feeling lonely and wanted some company.", "model_answer": "Susie", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F3", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F3"}
|
||
{"instance_id": "locomo_conv-26_q128", "cell": "agentic", "question": "What did Caroline find in her neighborhood during her walk?", "ground_truth": "a rainbow sidewalk", "context": "Session 14 (1:33 pm on 25 August, 2023):\nCaroline: I was out walking in my neighborhood when I came across this cool rainbow sidewalk for Pride Month. It was so vibrant and welcoming, I had to take a picture! It reminds us that love and acceptance are everywhere—even where we least expect it.", "model_answer": "a rainbow sidewalk", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-50_q010", "cell": "agentic", "question": "When did Calvin's place get flooded in Tokyo?", "ground_truth": "On a week before 16 May, 2023", "context": "Session 6 (11:50 am on 16 May, 2023):\nCalvin: Hey Dave, not everything has been going smoothly. I had an incident last week where my place got flooded, but thankfully, I managed to save my music gear and favorite microphone. It's been tough, but I'm staying positive and looking forward to getting everything fixed up.", "model_answer": "last week", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-50_q078", "cell": "agentic", "question": "What was the necklace Calvin received meant to remind him of?", "ground_truth": "why he keeps hustling as a musician", "context": "Session 4 (6:24 pm on 1 May, 2023):\nCalvin: Thanks, Dave! I got it from another artist as a gift - it's a great reminder of why I keep hustling as a musician!", "model_answer": "why he keeps hustling as a musician", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-47_q092", "cell": "agentic", "question": "What did John create for the charitable foundation that helped generate reports for analysis?", "ground_truth": "computer application on smartphones", "context": "Session 11 (5:00 pm on 11 May, 2022):\nJohn: Previously, this foundation used paper records and all inventory was recorded manually. I made an application that structured their work, and now everything they need for inventory is in one application on their smartphone.", "model_answer": "a software tool", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-30_q057", "cell": "agentic", "question": "What advice does Gina give to Jon about running a successful business?", "ground_truth": "build relationships with customers, create a strong brand image, stay positive", "context": "Session 7 (7:28 pm on 23 March, 2023):\nJon: Yeah, brand identity is key. Make sure yours stands out. Also be sure to build relationships with your customers – let them know you care. And don't forget to stay positive and motivate others. Your energy will be contagious!", "model_answer": "Focus on why you started, have faith in yourself, remind yourself of successes and progress, and have a good support system", "opus_verdict": "incorrect", "opus_failure_mode": "F3", "gpt_verdict": "incorrect", "gpt_failure_mode": "F5", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F4"}
|
||
{"instance_id": "locomo_conv-30_q029", "cell": "agentic", "question": "Which cities has Jon visited?", "ground_truth": "Paris, Rome", "context": "Session 2 (2:32 pm on 29 January, 2023):\nJon: Hey Gina! Thanks for asking. I'm on the hunt for the ideal spot for my dance studio and it's been quite a journey! I've been looking at different places and picturing how the space would look. I even found a place with great natural light! Oh, I've been to Paris yesterday! It was sooo cool. [image: a photo of a bathroom with a blue floor and a pink wall]\n\nSession 15 (10:04 am on 19 June, 2023):\nJon: Hey Gina, hope you're doing great! Still working on my biz. Took a short trip last week to Rome to clear my mind a little.", "model_answer": "unknown", "opus_verdict": "incorrect", "opus_failure_mode": "F1", "gpt_verdict": "incorrect", "gpt_failure_mode": "F1", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F1"}
|
||
{"instance_id": "locomo_conv-44_q066", "cell": "agentic", "question": "What type of dog was Andrew looking to adopt based on his living space?", "ground_truth": "smaller dog", "context": "Session 5 (10:47 am on 6 May, 2023):\nAndrew: Meeting all these adorable pups has been awesome! For those considering getting a pup, the size of living space and the exercise needs of the breed are important. For me, a person living in an apartment, a smaller dog would be best, but if one is active, consider getting one that loves to play and run.", "model_answer": "a smaller dog", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-44_q002", "cell": "agentic", "question": "What kind of indoor activities has Andrew pursued with his girlfriend?", "ground_truth": "boardgames, volunteering at pet shelter, wine tasting, growing flowers", "context": "Session 13 (3:52 pm on 27 July, 2023):\nAndrew: Hey Audrey! How are you? My GF and I just had a great experience volunteering at a pet shelter on Monday - it was so rewarding! We loved spending time with those cute animals and it gave us so much joy. It was so rewarding, it reminded me just how much I love them!\n\nSession 19 (5:53 pm on 24 September, 2023):\nAndrew: Yeah! They really do bring so much into our lives - it's amazing to watch them interact. Here's something I've been taking care of lately. Look at those flowers! [image: a photo of a balcony with a bunch of flowers on it]\n\nSession 23 (4:22 pm on 13 October, 2023):\nAndrew: Hey Audrey, it's been a busy week for me. Last Tuesday, my gf, Toby, and I had a really awesome night playing board games. It was really nice. What's been up with you lately? [image: a photography of a dog sitting at a chess board with a chess set in front of him]\n\nSession 25 (10:14 am on 24 October, 2023):\nAndrew: Hi Audrey! How have you been lately? My girlfriend and I went to this awesome wine tasting last weekend. It was great! We tried so many unique wines and learned a lot. I was surprised at how much I enjoyed it. A reminder to step out of the comfort zone!", "model_answer": "board games", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F2"}
|
||
{"instance_id": "locomo_conv-47_q046", "cell": "agentic", "question": "When did John and James meet at McGee's bar?", "ground_truth": "August 27, 2022", "context": "Session 21 (9:18 pm on 26 August, 2022):\nJohn: Great, then I agree! See you tomorrow at McGee's Pub!", "model_answer": "Tomorrow", "opus_verdict": "incorrect", "opus_failure_mode": "F2", "gpt_verdict": "incorrect", "gpt_failure_mode": "F2", "gemini_verdict_v5": "incorrect", "gemini_failure_mode_v5": "F3"}
|
||
{"instance_id": "locomo_conv-43_q076", "cell": "agentic", "question": "What kind of picture did Tim share as part of their Harry Potter book collection?", "ground_truth": "MinaLima's creation from the Harry Potter films", "context": "Session 2 (5:08 pm on 15 June, 2023):\nTim: Thanks! That picture is from MinaLima. They created all the props for the Harry Potter films, and I love their work. It's like having a piece of the wizarding world at home!", "model_answer": "A picture from MinaLima", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|
||
{"instance_id": "locomo_conv-41_q065", "cell": "agentic", "question": "What is John's main focus in local politics?", "ground_truth": "Improving education and infrastructure", "context": "Session 1 (11:01 am on 17 December, 2022):\nJohn: I'm passionate about improving education and infrastructure in our community. Those are my main focuses.", "model_answer": "Education and infrastructure", "opus_verdict": "correct", "opus_failure_mode": null, "gpt_verdict": "correct", "gpt_failure_mode": null, "gemini_verdict_v5": "correct", "gemini_failure_mode_v5": null}
|