World Alignment: unpaired cross-modal correspondence by relational identifiability

Method: scene states are sets of part states; relation fields are built
within each modality and are invariant to how each side labels its own
features; the cross-modal bridge is a coupling searched under an energy
that is a closed-form functional of one matrix; solving is spectral
initialisation followed by exact local refinement.

Evidence: in a procedurally generated closed world, blind recovery of a
hidden image-caption correspondence reaches 95.3% at 256 scenes against
0.39% chance, and the recovered pairs transfer to 200 held-out scenes at
93.0% exact retrieval with random-pair and shuffled-image controls at or
near chance. Cross-modal value correspondence is derived from disjoint
corpora rather than declared. On Visual Genome the field correlation
reaches 0.656 against the 0.9 that polynomial recovery needs, with the
deficit attributed away from segmentation and discretisation.

Protocol: no image-text pair enters any objective, optimiser,
initialisation, or model selection; hidden pairs score orderings only.

Co-Authored-By: Claude <noreply@anthropic.com>

1 files changed, 337 insertions, 0 deletions

diff --git a/artifacts/energy_prefix_evaluation.json b/artifacts/energy_prefix_evaluation.json
new file mode 100644
index 0000000..68d4e5c
--- /dev/null
+++ b/artifacts/energy_prefix_evaluation.json
@@ -0,0 +1,337 @@
+{
+  "samples": 100,
+  "mean_best_reference_unigram_f1": {
+    "initial": 0.24912132451795435,
+    "final": 0.22871332646020204,
+    "oracle": 0.5369529475719557
+  },
+  "energy_protocol": "No cross-modal map and no image-text pair is used by the energy or optimizer. Paired text is loaded only for trajectory and oracle diagnostics; the final step is fixed by CLI arguments.",
+  "prefix_training": "text only; no image features or image-text pairs",
+  "examples": [
+    {
+      "row": 25,
+      "references": [
+        "The man with pierced ears is wearing glasses and an orange hat.",
+        "A man with glasses is wearing a beer can crocheted hat.",
+        "A man with gauges and glasses is wearing a Blitz hat.",
+        "A man in an orange hat starring at something.",
+        "A man wears an orange hat and glasses."
+      ],
+      "initial": "Someone, wearing a white shirt and a black hat, is using a wooden spoon to make something with a mixture of wheat and rice.  It is possible",
+      "final": "Bumbsuckle pencils be in a bunch. \n*   *   *   *   *   *   *   *   *   *   *",
+      "oracle": "The man with the glasses is wearing an orange hat and wearing glasses."
+    },
+    {
+      "row": 34,
+      "references": [
+        "A black and white dog is running in a grassy garden surrounded by a white fence.",
+        "A Boston Terrier is running on lush green grass in front of a white fence.",
+        "A black and white dog is running through the grass.",
+        "A dog runs on the green grass near a wooden fence.",
+        "A Boston terrier is running in the grass."
+      ],
+      "initial": "A young child is playing with a green and yellow toy in a suburban area. The child is sitting on a bench and is using a sprinkler to water the",
+      "final": "A small child is playing with a bubble wand. The child is pushing the wand. The child is playing in the water.",
+      "oracle": "A black and white dog is running in a grassy area surrounded by a fence.  Captioned: \"Running in the Park\".  Copyrighted. "
+    },
+    {
+      "row": 51,
+      "references": [
+        "A young female student performing a downward kick to break a board held by her Karate instructor.",
+        "Girl about to kick a piece of wood in half while karate instructor holds it",
+        "A girl kicking a stick that a man is holding in tae kwon do class.",
+        "A girl in karate uniform breaking a stick with a front kick.",
+        "A girl breaking boards by using karate."
+      ],
+      "initial": "Three people in a circular dome are held by a mic and a camera, with a man in a halo behind them.",
+      "final": "A boy is standing in front of an ornately-wooden, glass-enclosed, large, white, and ornate glass dome. He is holding",
+      "oracle": "A young female karate student attempting to balance her leg over a wall while practicing her kicks."
+    },
+    {
+      "row": 89,
+      "references": [
+        "Five snowmobile riders all wearing helmets and goggles line up in a snowy clearing in a forest in front of their snowmobiles; they are all wearing black snow pants and from left to right they are wearing a black coat, white coat, red coat, blue coat, and black coat.",
+        "Five people wearing winter jackets and helmets stand in the snow, with snowmobiles in the background.",
+        "Five people wearing winter clothing, helmets, and ski goggles stand outside in the snow.",
+        "A group of snowmobile riders gather in the snow.",
+        "Group gathered to go snowmobiling."
+      ],
+      "initial": "A group of men in red coats sit around a table in a dimly lit room and one of them is wearing a hat. The men are all wearing red",
+      "final": "Two women in black hats talking behind each other.  What kind of hat are they wearing?",
+      "oracle": "Three snowmobiles are parked in a line, with riders wearing helmets and white helmets in the front, and white helmets in the back. One of the snow"
+    },
+    {
+      "row": 97,
+      "references": [
+        "Two men sitting on the roof of a house while another one stands on a ladder.",
+        "Two men on a rooftop while another man stands atop a ladder watching them",
+        "Three men, one on a ladder, work on a roof.",
+        "People are fixing the roof of a house.",
+        "Three men are working on a roof."
+      ],
+      "initial": "Three people, one woman, and one man, all hold a glass of beer and look at a group of people. One of the women is holding a camera",
+      "final": "Four people are walking in the street. One is carrying a red flag. The rest are looking at the street. \n–Oli)",
+      "oracle": "Two men sitting on a roof while one of them works on a roof of a building."
+    },
+    {
+      "row": 124,
+      "references": [
+        "A bride in a light pink dress poses for a picture with male relatives and is being photographed by a man in a cream shirt with white pants.",
+        "A man in light colored clothing photographs a group of men wearing dark suits and hats standing around a woman dressed in a strapless gown.",
+        "A photographer takes a picture of a group of one girl in a pink dress and 10 boys in suits and hats.",
+        "A man photographs a woman in a pink dress and a throng of men in suits.",
+        "Man taking a photograph of a well dressed group of teens."
+      ],
+      "initial": "A brown and white dog walks in the grass with his hair in the wind. The dog is running with a stick in his mouth. License: ©20",
+      "final": "A man in a black suit and white checkered shirt is standing in front of a mirror while looking at his face in a mirror behind him. The man is",
+      "oracle": "A bride in a white dress is being photographed with a groom in a black suit and a photographer in a white shirt. The bride is posed for the camera while"
+    },
+    {
+      "row": 133,
+      "references": [
+        "A group of 11 people in winter wear such as beanies, skiing jackets, gloves and backpacks are standing in snow paddles outside a house made of ice blocks while a person in front of the door seems to be leading them.",
+        "A group of people wearing snowshoes, and dressed for winter hiking, is standing in front of a building that looks like it's made of blocks of ice.",
+        "The people are quietly listening while the story of the ice cabin was explained to them.",
+        "A group of people standing in front of an igloo.",
+        "Several students waiting outside an igloo."
+      ],
+      "initial": "Two women dressed in red and white blankets leaped over a wooden ramp. They are covered in brown fur. \nClassification: Natural. \nUpper Right Side",
+      "final": "Two women in hairbands wearing headwear fronting surfers.",
+      "oracle": "A group of people wearing snow gear, gloves and hats are standing in front of a building, one person is wearing a scarf and a hat, another is wearing"
+    },
+    {
+      "row": 134,
+      "references": [
+        "A baseball player in red leaps into the air to avoid a player in blue, who is shortly behind him as he catches the ball, while other blue team members look on in the background.",
+        "A boy in a red uniform is attempting to avoid getting out at home plate, while the catcher in the blue uniform is attempting to catch him.",
+        "A baseball player coming into home plate with knee held high and the catcher right in front of him waiting for the ball.",
+        "A baseball player attempting to score a run before the catcher tags the player out.",
+        "A baseball catcher trying to tag a base runner in a baseball game."
+      ],
+      "initial": "The man is doing a trick on his bicycle in the city. Description of photo is unclear. Rating is 3 out of 5. Title is not clear",
+      "final": "A red-looked man is turning a wheel while holding a red and black car. Title: Red Wheel Turns Red Car.",
+      "oracle": "A baseball player in red is caught in midair as another player in black and white prepares to hit the ball, while the latter is in the background. The"
+    },
+    {
+      "row": 137,
+      "references": [
+        "A man wearing a hat and a white shirt is cleaning windows.",
+        "A man in a white shirt stands high up on scaffolding",
+        "A man stands on boards on top of a huge ladder.",
+        "Man works on top of scaffolding.",
+        "A guy works on a building."
+      ],
+      "initial": "Man in red and black dress is using a camera while eating and walking.  He is wearing a blue shirt.  He is also wearing a pair of sunglasses",
+      "final": "Three vehicles are walking alongside a white border.  Distribution of characters 1/2.  Closely spaced.  Edges of the border are visible",
+      "oracle": "A man wearing a hat and glasses is cleaning a room."
+    },
+    {
+      "row": 199,
+      "references": [
+        "A man wearing a reflective vest sits on the sidewalk and holds up pamphlets with bicycles on the cover.",
+        "Man in bright yellow vest displays bicycle safety information on street.",
+        "A man and woman talking are interrupted and a man covers his face.",
+        "A man in a vest is sitting in a chair and holding magazines.",
+        "A construction worker holds a sign about bicycling."
+      ],
+      "initial": "Couple in the city, kissing, with a train in the background, in a car.  Also, a girl in a pink shirt, in the background",
+      "final": "A woman running in a dark colored shirt and blue shorts on a sunny day. Source: unknown. Tags: #soccer #women #running . Category:",
+      "oracle": "A man wearing a yellow jacket holds a sign on the sidewalk and sits on a bench with the words \"No Smoking\" on it. People walk by the sign"
+    },
+    {
+      "row": 219,
+      "references": [
+        "A mother decides to take her child on a piggyback ride outside their apartment complex.",
+        "A baby boy in a blue and white striped shirt is sitting on his mother's shoulders.",
+        "A mother and her young song enjoying a beautiful day outside.",
+        "A young woman is giving a baby a ride on her shoulders.",
+        "A woman gives a small child a piggyback ride."
+      ],
+      "initial": "A man sits near the edge of a rectangular area that is surrounded by a large rock wall.",
+      "final": "A girl is holding a bowl of food and a basket of food on her head. She is wearing a hat and a coat.",
+      "oracle": "A mother takes her daughter to the park in a new baby seat."
+    },
+    {
+      "row": 222,
+      "references": [
+        "A male volleyball player in red trunks tries to block the shot of the opposing player during a professional volleyball game.",
+        "Men playing volleyball, with one player missing the ball but hands still in the air.",
+        "A group of spectators watch a men's sand volleyball game.",
+        "Volleyball players compete on a sunny day at the beach.",
+        "Men playing volleyball in the sand."
+      ],
+      "initial": "A student girl, a man with a black jacket and a girl with a hat, are leaning on a window with a seat. The girl is looking at a",
+      "final": "Man is wearing a red shirt and a black and white mace, \nmaking a fast, rapid, and precise motion with a sword. \n\nWhat is the",
+      "oracle": "A male player in a volleyball team prepares to block a shot from the opposing team during a match."
+    },
+    {
+      "row": 228,
+      "references": [
+        "A woman wearing a pink shirt and red apron stands in her restaurant holding food and surrounded by bagged breads, a microwave and the chalkboard menu up on the wall.",
+        "Woman wearing a yellow hat, pink shirt, and red apron is holding food in a kitchen.",
+        "Employee in pink shirt smiles for camera inside a restaurant kitchen.",
+        "A woman holding a bowl of food in a kitchen.",
+        "Women in store happily holding some food."
+      ],
+      "initial": "A boy smiles while sitting in front of a blue and white blender with a camera in front of him.",
+      "final": "Three children play with instruments while the teacher discusses with them. Source: Image by Kiki123456. Creative Commons Attribution-ShareAlike",
+      "oracle": "A woman wearing a white shirt and apron stands in front of a table with a large tray of food and drinks, holding a glass of water and a plate"
+    },
+    {
+      "row": 246,
+      "references": [
+        "An Asian man in spectacles and a t-shirt preparing some food.",
+        "A man working diligently while sitting at a table.",
+        "Man sitting using tool at a table in his home.",
+        "Indian man whittles in his own home.",
+        "A man is sharpening a knife."
+      ],
+      "initial": "An older man lays on the blue sand and looks at the ground. He uses a shovel. He is wearing a blue cap. \n Does the word \"lay",
+      "final": "Dons of a school play go down the stairs, looking at each other.  They are in a darkened auditorium.  They are interacting with the",
+      "oracle": "An Asian man in a t-shirt and shorts trying to purchase some food. Also wearing a hat. \nTags: asian, shopping, t-shirt\n\nAssistant:"
+    },
+    {
+      "row": 259,
+      "references": [
+        "Spelunkers pose inside a rock cavern while bathed in sunlight from the surface.",
+        "Shaft of light in a cave shows three spelunkers",
+        "Three rock climbers are posing on big rocks.",
+        "People stand inside a rock dome.",
+        "Three people sit in a cave."
+      ],
+      "initial": "A blue dressed man playing a banjo in a dark blue and yellow room. \nDoes this next sentence follow, given the preceding text?\nA man is playing",
+      "final": "A whileon with a whileon while artfully covered in a snow cover.",
+      "oracle": "Holes appear in the darkness as two miners explore a dark cavern. A dark room with a water tank is visible in the background."
+    },
+    {
+      "row": 279,
+      "references": [
+        "A girl in a polka dotted blue jean dress walks barefoot on a balance beam.",
+        "A girl in a jean dress is walking along a raised balance beam.",
+        "The little girl is walking a beam in gymnastics class.",
+        "A little girl balances on a gymnastics beam.",
+        "A little girl is walking on a balance beam."
+      ],
+      "initial": "An 11-year-old boy in his t-shirt and jeans sitting on the ice rink in the middle of the street. He is holding his ice r",
+      "final": "矮小的儿童佩戴口罩和头盔，正进行手部动作训练。他或她正在使用手部工具。",
+      "oracle": "A girl in a pink polka dot dress walks on a blue and white striped floor. The girl is in a pink polka dot dress. \nDoes this"
+    },
+    {
+      "row": 304,
+      "references": [
+        "A couple is sitting on the sand with their feet in the water, and they are shaking hands.",
+        "A man and woman wearing sunglasses sit halfway in the water.",
+        "A boy and a girl holding hands while sitting on a shore.",
+        "A couple with sunglasses are laying on the beach.",
+        "A blond holding hands with a guy in the sand."
+      ],
+      "initial": "A military officer is wearing a military-style dress and a military-style headband while a group of men are holding a flag.",
+      "final": "A man is wearing a red and white cloth while walking with a large amount of material.",
+      "oracle": "A couple is sitting on the beach, and the person in the sand is holding their hands.  They are standing in the water.  They are also wearing"
+    },
+    {
+      "row": 318,
+      "references": [
+        "A woman standing with 3 other people in a store with two tables, some shelves with coffee and tea for sale, and a refrigerated drink case.",
+        "A woman in a gray sweater and black baseball cap is standing in line at a shop.",
+        "A woman in a hat waits to be served at a store.",
+        "Young people are standing in the store.",
+        "A woman in a cap at a coffee shop."
+      ],
+      "initial": "A large brown dog is walking in a shaded area while a small brown animal is hopping on its back. Caption: A young boy is watching the dog.\nLarge",
+      "final": "the kids are trying to get the ball from the other team while the one in the red shirt tries to stop them from getting it.  At the same time",
+      "oracle": "A woman standing in a store with 2 rows of stands, a woman in a blue shirt and a man in a blue shirt standing in the stands. There"
+    },
+    {
+      "row": 340,
+      "references": [
+        "The person has a striped shirt on and is holding on to a rope on a mountain.",
+        "A kid rock climbing against the backdrop of a green valley",
+        "The person in the striped shirt is mountain climbing.",
+        "A woman in a striped shirt climbs up a mountain.",
+        "A young man climbs a rocky hill."
+      ],
+      "initial": "A small, brown dog, wearing a blue shirt, chases a smaller, gray dog. The smaller dog is running away. The larger dog is also running",
+      "final": "A surfer is using a board to catch the wind in the ocean.",
+      "oracle": "The person on the leash is holding on to a piece of rope on a sidewalk with a sign on it. He is on a balcony."
+    },
+    {
+      "row": 395,
+      "references": [
+        "Two young attractive women in front of a store drop coins for street performers.",
+        "Two women enjoy the entertainment of street performers.",
+        "Two men pretend to be statutes while women look on.",
+        "Two young women stop and watch street performers.",
+        "Pedestrians interact with street artists."
+      ],
+      "initial": "A baseball player in a uniform plays safely as a catcher waits for the ball to be thrown. He is wearing a black uniform and a baseball cap. The catcher",
+      "final": "Baseball player swinging at the base of the red bat with the red and white baton in his hand, as the player on the left swings at the same",
+      "oracle": "Two young women in front of a street market display of clothes and money for two young people. Behind them are two young people in front of a bank. Portrait"
+    },
+    {
+      "row": 434,
+      "references": [
+        "A group of people standing on the lawn in front of a building.",
+        "Many people in blue jeans stand in front of a white church.",
+        "A large group of people stand outside of a church.",
+        "Family members standing outside a home.",
+        "People standing outside of a building."
+      ],
+      "initial": "There is a little boy wearing a blue shirt and a green helmet. He is playing with a small toy.",
+      "final": "A man holding a bottle of beer with one arm and a hammer in the other.",
+      "oracle": "A group of people standing on the front of a building in a city."
+    },
+    {
+      "row": 472,
+      "references": [
+        "A young girl wearing a blue shirt marching in a band playing a trumpet",
+        "Girl wearing blue shirt and black shorts plays trumped outside.",
+        "A teenager plays her trumpet on the field at a game.",
+        "A girl playing trumpet in a marching band.",
+        "Girl playing the trumpet in a marching band"
+      ],
+      "initial": "A man wearing a blue shirt and blue jeans is sitting on the floor with his mouth closed and a blue baseball on his head.",
+      "final": "A low resolution picture of a red and black looking dog looking at a yellow looking bowl. \n    Copyrighted by the author. Licensed under the Creative Commons Attribution",
+      "oracle": "A young girl wearing a red and white striped shirt playing a marching band in a line of young people."
+    },
+    {
+      "row": 487,
+      "references": [
+        "Person in a brown sweatshirt does a flip in midair over a trampoline with a river and electrical lines in the background.",
+        "A young adult is doing a back flip on a trampoline near a lake.",
+        "A woman is doing flips on a trampoline in front of a lake.",
+        "A woman does a somersault on a trampoline on the beach.",
+        "Trampolines are a fun way to exercise."
+      ],
+      "initial": "A little girl with a pink shirt and a green coat standing next to a toy on a table with a green box in her hands.",
+      "final": "Two men, one wearing a red shirt and hiking gear, and the other wearing a black and white striped shirt and hiking gear, are trying to climb a mountain",
+      "oracle": "Man in a t-shirt jumps over a net with a skateboard in the background with a body of water in the background. A cityscape in the background. In"
+    },
+    {
+      "row": 520,
+      "references": [
+        "A man standing in a bar with a neon beer sign hanging on the wall",
+        "A man stands next to three video game machines and a beer sign.",
+        "A man stands next to a bank of computer gambling machines.",
+        "A man is standing by a group of video games in a bar.",
+        "A man stands next to three video machines."
+      ],
+      "initial": "The couple is sitting on a lawn chair with a yellow wedding cake in the center. The path is to the left.  Copyright © 2014",
+      "final": "A young, white child, in a red, white, and blue uniform, plays in a snow-covered, grassy park, surrounded by green grass, a",
+      "oracle": "A man standing in a pub with a neon sign hanging from his arm."
+    },
+    {
+      "row": 539,
+      "references": [
+        "A woman in a green tank top looking at a drill while a crowd looks on.",
+        "A woman using a power drill in front of a group of people.",
+        "A woman uses a drill while another man takes her picture.",
+        "A woman, in a teal tank top, holding a power drill.",
+        "Spectators about to watch a woman use a drill."
+      ],
+      "initial": "A man in a green jacket, black pants, and a yellow shirt, is doing a jigsaw, in a pattern of a square, with a black and",
+      "final": "A man is making music while he is in the right position. \nRelated Questions\nRelated Questions\nRelated Questions\nRelated Questions\nRelated Questions\nRelated Questions\n",
+      "oracle": "A woman in a green shirt looking at a drill while a man looks on."
+    }
+  ]
+}
\ No newline at end of file