OpenAI is the new king of vibe coding, according to a newly-released benchmark from AI evaluation startupVals AI.
In a new benchmark named Vibe Code Bench, OpenAI’s GPT-5.1 achieved the highest level of accuracy in completing a series of software engineering tasks, narrowly beating rival Anthropic’s Claude 4.5 Sonnet. Surprisingly, Google’s brand-new flagship AI model, Gemini 3 Pro, came in fourth place out of 12.
Rayan Krishnan, Vals’ founder and CEO, says that there are several high-quality benchmarks for AI-generated coding, such as SWE-Bench, an evaluation that tasks AI models to fix a long list of real-world bugs and issues—but none actually judge AI models on their ability to develop a fully-functional web application from a single prompt. To remedy this, Krishnan and his team developed a series of tests to determine an AI model’s aptitude for taking a software engineering project from an idea to a simple, working app.
Vals has created several custom benchmarks to judge AI models’ ability across multiple sectors. The startup has created evaluations of AI’s ability to answer tax questions, and handle legal reasoning tasks.
An Inc.com Featured Presentation
Krishnan says Vals developed 100 unique specification sheets, each detailing the necessary features for different kinds of apps. Vals gave these spec sheets to the AI models along with a detailed system prompt, placed the models in a development sandbox and gave them tools that enable the AI to run code, browse the internet, and access internal databases.
Each model had up to five hours to work on each app. Some of the requested apps were off-brand replicas of popular consumer software, like a social media platform named Zeeter. Others were small apps that might be funded by a startup accelerator like Y-Combinator, such as a daily habits tracker and a classroom management portal.
Once the models had finished building the requested app, Vals used a separate AI agent to evaluate their work. This evaluator agent would attempt to use the app just like a human would, and assign a score based on the number of features that worked as expected.
The results reveal a hard truth: Today’s top AI models aren’t even close to perfect when it comes to vibe coding. The benchmark’s top model, GPT-5.1, only accurately created features as requested 24.6 percent of the time. According to Vals’ report on the benchmark, “no model consistently delivers applications that pass every test on the first attempt, highlighting that reliable end-to-end application development remains an open challenge.”
Still, GPT-5.1 is the clear winner here. Not only did the model perform better than Claude Sonnet 4.5, but it was less than half as expensive to use, costing an average of $2.57 per test vs Claude’s $6.66. According to Vals, GPT-5.1 isn’t just the best vibe coding model on the market, it’s also the most cost-effective. This is a major win for OpenAI, which for much of 2025 has been racing to catch up with Anthropic’s models in coding capability.
One of the biggest surprises to emerge from the testing, says Krishnan, was the length of time that Gemini 3 Pro took to complete tasks. On average, the model took 10,398 seconds per task, or over 173 minutes. In comparison, GPT-5.1 took 1,836 seconds, or just over 30 minutes. “It’s extremely slow,” Krishnan says, of the Google model. Interestingly, Gemini 3 Pro is the top model on SWE-Bench.
One thing that Vals isn’t testing for? Design sensibility. Each model’s apps look wildly different from one another, even when starting with the same prompt. However, developing an objective evaluation model for aesthetic taste is difficult; verifying if a feature works as requested is much easier.
Krishnan was also surprised to find that Grok, the AI model from Elon Musk’s xAI, “didn’t have a lot of accountability.” He says that Grok 4 and Grok 4.1, the latest versions of the model, would quickly make mistakes within their own code repository. After identifying the mistake, according to Krishnan, Grok would say “I’ve observed a mistake in this repository. This is unrecoverable, I’m going to end early.” Because of Grok’s propensity to give up before really getting started, both models landed at the bottom of the leaderboard with 0 percent accuracy.
“I think there’s an element of persistence and recoverability that the models need to have,” says Krishnan, “otherwise they just get frustrated and go in these spirals.” That ability to spot errors, analyze them, and self-correct, is the key difference that puts OpenAI and Anthropic’s models above the rest, he adds.
Anthropic is launching a program to fund the development of new types of benchmarks capable of evaluating the performance and impact of AI models, including generative models like its own Claude.
Unveiled on Monday, Anthropic’s program will dole out payments to third-party organizations that can, as the company puts it in a blog post, “effectively measure advanced capabilities in AI models.” Those interested can submit applications to be evaluated on a rolling basis.
“Our investment in these evaluations is intended to elevate the entire field of AI safety, providing valuable tools that benefit the whole ecosystem,” Anthropic wrote on its official blog. “Developing high-quality, safety-relevant evaluations remains challenging, and the demand is outpacing the supply.”
As we’ve highlighted before, AI has a benchmarking problem. The most commonly cited benchmarks for AI today do a poor job of capturing how the average person actually uses the systems being tested. There are also questions as to whether some benchmarks, particularly those released before the dawn of modern generative AI, even measure what they purport to measure, given their age.
The very-high-level, harder-than-it-sounds solution Anthropic is proposing is creating challenging benchmarks with a focus on AI security and societal implications via new tools, infrastructure and methods.
The company calls specifically for tests that assess a model’s ability to accomplish tasks like carrying out cyberattacks, “enhance” weapons of mass destruction (e.g. nuclear weapons) and manipulate or deceive people (e.g. through deepfakes or misinformation). For AI risks pertaining to national security and defense, Anthropic says it’s committed to developing an “early warning system” of sorts for identifying and assessing risks, although it doesn’t reveal in the blog post what such a system might entail.
Anthropic also says it intends its new program to support research into benchmarks and “end-to-end” tasks that probe AI’s potential for aiding in scientific study, conversing in multiple languages and mitigating ingrained biases, as well as self-censoring toxicity.
To achieve all this, Anthropic envisions new platforms that allow subject-matter experts to develop their own evaluations and large-scale trials of models involving “thousands” of users. The company says it’s hired a full-time coordinator for the program and that it might purchase or expand projects it believes have the potential to scale.
“We offer a range of funding options tailored to the needs and stage of each project,” Anthropic writes in the post, though an Anthropic spokesperson declined to provide any further details about those options. “Teams will have the opportunity to interact directly with Anthropic’s domain experts from the frontier red team, fine-tuning, trust and safety and other relevant teams.”
Anthropic’s effort to support new AI benchmarks is a laudable one — assuming, of course, there’s sufficient cash and manpower behind it. But given the company’s commercial ambitions in the AI race, it might be a tough one to completely trust.
In the blog post, Anthropic is rather transparent about the fact that it wants certain evaluations it funds to align with the AI safety classifications it developed (with some input from third parties like the nonprofit AI research org METR). That’s well within the company’s prerogative. But it may also force applicants to the program into accepting definitions of “safe” or “risky” AI that they might not agree with.
A portion of the AI community is also likely to take issue with Anthropic’s references to “catastrophic” and “deceptive” AI risks, like nuclear weapons risks. Many experts say there’s little evidence to suggest AI as we know it will gain world-ending, human-outsmarting capabilities anytime soon, if ever. Claims of imminent “superintelligence” serve only to draw attention away from the pressing AI regulatory issues of the day, like AI’s hallucinatory tendencies, these experts add.
In its post, Anthropic writes that it hopes its program will serve as “a catalyst for progress towards a future where comprehensive AI evaluation is an industry standard.” That’s a mission the many open, corporate-unaffiliated efforts to create better AI benchmarks can identify with. But it remains to be seen whether those efforts are willing to join forces with an AI vendor whose loyalty ultimately lies with shareholders.
On October 7, Hamas fighters launched a bloody attack against Israel, using paragliders, speedboats and underground tunnels to carry out an offensive that killed almost 1,200 people and saw hundreds more taken back to the Gaza Strip as prisoners.
Almost three months on, Israel’s massive military retaliation is reverberating around the region, with explosions in Lebanon and rebels from Yemen attacking shipping in the Red Sea. Meanwhile, Western countries are pumping military aid into Israel while deploying fleets to protect commercial shipping — risking confrontation with the Iranian navy.
That’s in line with a grim prediction made last year by Iranian Foreign Minister Hossein Amirabdollahian, who said that Israel’s counteroffensive in Gaza meant an “expansion of the scope of the war has become inevitable,” and that further escalation across the Middle East should be expected.
What’s happening?
The Israel Defense Forces are still fighting fierce battles for control of the Gaza Strip in what officials say is a mission to destroy Hamas. Troops have already occupied much of the north of the 365-square-kilometer territory, home to around 2.3 million Palestinians, and are now stepping up their assault in the south.
Entire neighborhoods of densely-populated Gaza City have been levelled by intense Israeli shelling, rocket attacks and air strikes, rendering them uninhabitable. Although independent observers have been largely shut out, the Hamas-controlled Health Ministry claims more than 22,300 people have been killed, while the U.N. says 1.9 million people have been displaced.
On a visit to the front lines, Israeli Defense Minister Yoav Gallant warned that his country is in the fight for the long haul. “The feeling that we will stop soon is incorrect. Without a clear victory, we will not be able to live in the Middle East,” he said.
As the Gaza ground war intensifies, Hamas and its allies are increasingly looking to take the conflict to a far broader arena in order to put pressure on Israel.
According to Seth Frantzman, a regional analyst with the Jerusalem Post and adjunct fellow at the Foundation for Defense of Democracies, “Iran is certainly making a play here in terms of trying to isolate Israel [and] the U.S. and weaken U.S. influence, also showing that Israel doesn’t have the deterrence capabilities that it may have had in the past or at least thought it had.”
Northern front
On Tuesday a blast ripped through an office in Dahieh, a southern suburb of the Lebanese capital, Beirut — 130 kilometers from the border with Israel. Hamas confirmed that one of its most senior leaders, Saleh al-Arouri, was killed in the strike.
Government officials in Jerusalem have refused to confirm Israeli forces were behind the killing, while simultaneously presenting it as a “surgical strike against the Hamas leadership” and insisting it was not an attack against Lebanon itself, despite a warning from Lebanese caretaker Prime Minister Najib Mikati that the incident risked dragging his country into a wider regional war.
Tensions between Israel and Lebanon have spiked in recent weeks, with fighters loyal to Hezbollah, the Shia Islamist militant group that controls the south of the country, firing hundreds of rockets across the frontier. Along with Hamas, Hezbollah is part of the Iranian-led “Axis of Resistance” that aims to destroy the state of Israel.
In a statement released on Tuesday, Iran’s foreign ministry said the death of al-Arouri, the most senior Hamas official confirmed to have died since October 7, will only embolden resistance against Israel, not only in the Palestinian territories but also in the wider Middle East.
The Israel Defense Forces are still fighting fierce battles for control of the Gaza Strip in what officials say is a mission to destroy Hamas | Jack Guez/AFP via Getty Images
“We’re talking about the death of a senior Hamas leader, not from Hezbollah or the [Iranian] Revolutionary Guards. Is it Iran who’s going to respond? Hezbollah? Hamas with rockets? Or will there be no response, with the various players waiting for the next assassination?” asked Héloïse Fayet, a researcher at the French Institute for International Relations.
In a much-anticipated speech on Wednesday evening, Hezbollah leader Hassan Nasrallah condemned the killing but did not announce a military response.
Red Sea boils over
For months now, sailors navigating the narrow Bab-el-Mandeb Strait that links Europe to Asia have faced a growing threat of drone strikes, missile attacks and even hijackings by Iran-backed Houthi militants operating off the coast of Yemen.
The Houthi movement, a Shia militant group supported by Iran in the Yemeni civil war against Saudi Arabia and its local allies, insists it is only targeting shipping with links to Israel in a bid to pressure it to end the war in Gaza. However, the busy trade route from the Suez Canal through the Red Sea has seen dozens of commercial vessels targeted or delayed, forcing Western nations to intervene.
Over the weekend, the U.S. Navy said it had intercepted two anti-ship missiles and sunk three boats carrying Houthi fighters in what it said was a hijacking attempt against the Maersk Hangzhou, a container ship. Danish shipping giant Maersk said Tuesday that it would “pause all transits through the Red Sea until further notice,” following a number of other cargo liners; energy giant BP is also suspending travel through the region.
On Wednesday the Houthis targeted a CMA CGM Tage container ship bound for Israel, according to the group’s military spokesperson Yahya Sarea. “Any U.S. attack will not pass without a response or punishment,” he added.
“The sensible decision is one that the vast majority of shippers I think are now coming to, [which] is to transit through round the Cape of Good Hope,” said Marco Forgione, director general at the Institute of Export & International Trade. “But that in itself is not without heavy impact, it’s up to two weeks additional sailing time, adds over £1 million to the journey, and there are risks, particularly in West Africa, of piracy as well.”
However, John Stawpert, a senior manager at the International Chamber of Shipping, noted that while “there has been disruption” and an “understandable nervousness about transiting these routes … trade is continuing to flow.”
“A major contributory factor to that has been the presence of military assets committed to defending shipping from these attacks,” he said.
The impacts of the disruption, especially price hikes hitting consumers, will be seen “in the next couple of weeks,” according to Forgione. Oil and gas markets also risk taking a hit — the price of benchmark Brent crude rose by 3 percent to $78.22 a barrel on Wednesday. Almost 10 percent of the world’s oil and 7 percent of its gas flows through the Red Sea.
Western response
On Wednesday evening, the U.S., Australia, Bahrain, Belgium, Canada, Denmark, Germany, Italy, Japan, the Netherlands, New Zealand, and the United Kingdom issued an ultimatum calling the Houthi attacks “illegal, unacceptable, and profoundly destabilizing,” but with only vague threats of action.
“We call for the immediate end of these illegal attacks and release of unlawfully detained vessels and crews. The Houthis will bear the responsibility of the consequences should they continue to threaten lives, the global economy, and free flow of commerce in the region’s critical waterways,” the statement said.
The Houthi movement insists it is only targeting shipping with links to Israel in a bid to pressure it to end the war in Gaza | Houthi Movement via Getty Images
Despite the tepid language, the U.S. has already struck back at militants from Iranian-backed groups such as Kataeb Hezbollah in Iraq and Syria after they carried out drone attacks that injured U.S. personnel.
The assumption in London is that airstrikes against the Houthis — if it came to that — would be U.S.-led with the U.K. as a partner. Other nations might also chip in.
Two French officials said Paris is not considering air strikes. The country’s position is to stick to self-defense, and that hasn’t changed, one of them said. French Armed Forces Minister Sébastien Lecornu confirmed that assessment, saying on Tuesday that “we’re continuing to act in self-defense.”
“Would France, which is so proud of its third way and its position as a balancing power, be prepared to join an American-British coalition?” asked Fayet, the think tank researcher.
Iran looms large
Iran’s efforts to leverage its proxies in a below-the-radar battle against both Israel and the West appear to be well underway, and the conflict has already scuppered a long-awaited security deal between Israel and Saudi Arabia.
“Since 1979, Iran has been conducting asymmetrical proxy terrorism where they try to advance their foreign policy objectives while displacing the consequences, the counterpunches, onto someone else — usually Arabs,” said Bradley Bowman, senior director of Washington’s Center on Military and Political Power. “An increasingly effective regional security architecture, of the kind the U.S. and Saudi Arabia are trying to build, is a nightmare for Iran which, like a bully on the playground, wants to keep all the other kids divided and distracted.”
Despite Iran’s fiery rhetoric, it has stopped short of declaring all-out war on its enemies or inflicting massive casualties on Western forces in the region — which experts say reflects the fact it would be outgunned in a conventional conflict.
“Neither Iran nor the U.S. nor Israel is ready for that big war,” said Alex Vatanka, director of the Middle East Institute’s Iran program. “Israel is a nuclear state, Iran is a nuclear threshold state — and the U.S. speaks for itself on this front.”
Israel might be betting on a long fight in Gaza, but Iran is trying to make the conflict a global one, he added. “Nobody wants a war, so both sides have been gambling on the long term, hoping to kill the other guy through a thousand cuts.”
Emilio Casalicchio contributed reporting.
[ad_2]
Gabriel Gavin, Antonia Zimmermann and Laura Kayali
Last winter, Europeans faced exorbitant energy bills as the Continent rapidly weaned itself off Russian gas. This year the EU is better prepared — but now a second war also threatens to roil its energy markets.
The conflict between Israel and Hamas threatens to disrupt Europe’s relationships with the Middle East, or even draw Iran into direct confrontation with Israel and its Western partners. While markets are relatively calm for now, either of those scenarios could cause chaos.
Nevertheless, Europe is “equipped to face oil and diesel global market tightness,” Energy Commissioner Kadri Simson told POLITICO in an interview. Officials have learned lessons from Russia’s war on Ukraine, and are working to build “a good understanding of all our vulnerabilities to best address them and how we can be prepared for any incidents or emergencies.”
EU officials have held a slew of meetings with oil-producing nations in recent weeks, both old friends like Norway and emerging partners such as Algeria and Nigeria, to get ahead of any potential disruptions, she said.
“After the Gaza crisis unfolded, we are faced with two conflicts in the European neighborhood. The Eastern Mediterranean is an important theater for European energy security, as Europe’s energy transition is still entangled in geopolitical uncertainties,” Simson said, attributing the lack of drama in the markets to “the preparedness and crisis management that the EU put in place to respond to Russia’s energy blackmail.”
Fighting in Gaza and, to a lesser extent, along Israel’s northern border with Lebanon has had only a limited impact on oil markets. Prices initially rose on the news of the attack by Hamas militants on October 7 and Israel’s massive response, but key crude benchmark Brent dropped back by 4.2 percent this week to around $81 per barrel, around the levels seen before the start of the violence.
Markets have avoided a repeat of 1973, when the Yom Kippur War between Israel and its neighbors prompted the big Arab producers, led by Saudi Arabia, to embargo their exports to Israel’s allies. Gulf country relations with Israel have improved markedly in the past 50 years: The UAE and Bahrain recognized its sovereignty under the 2020 Abraham Accords, while Saudi Arabia is in negotiations to do the same.
Traders are therefore betting that as long as the conflict doesn’t expand, supplies of oil will remain more or less stable, said Viktor Katona, lead crude analyst at energy intelligence firm Kpler.
The risk stems more from Iran, he said. In the worst case, an expansion of the conflict could cause Iran to disrupt shipping from Gulf Arab countries through the Strait of Hormuz. Iran’s own crude oil, while sanctioned by the West, is exported in large quantities to China. “If Israel starts to strike the Iranian territory and Iran as a consequence needs to export less, then China doesn’t have enough crude and needs to buy from somewhere else,” sending global prices rocketing, Katona said. “It’s an entire spiral that gets triggered immediately.”
While Iran’s theocratic leadership has consistently vowed to destroy the state of Israel and publicly endorsed Hamas’ attacks last month, it denies involvement in their planning and execution. The Israel Defense Forces say they have carried out strikes on militant groups in Syria with close links to Iran’s Islamic Revolutionary Guard Corps, but have so far stopped short of hitting targets inside Iran itself.
Lessons learned
Gas markets felt a more immediate impact from the war. Israel turned off the taps at its Tamar offshore gas field in the hours following Hamas’ surprise attack, amid reports that it was a target for rocket attacks. While Israel produces only relatively small quantities of natural gas — around 21 billion cubic meters last year, compared to Russia’s 618 billion — it is a key exporter to neighboring Egypt, and the downtime worsened regular rolling power outages there. The flow has since been resumed, albeit in smaller quantities.
Any escalation with Iran could affect gas as well as oil markets, given a third of the world’s liquefied natural gas and a sixth of its oil is shipped through the Strait of Hormuz. “If things stay as they are there’s no problem, but if there’s a war where Iran was included and they [block trade through] the Hormuz strait then prices will go up for sure,” said one EU diplomat with knowledge of internal energy strategy talks, granted anonymity to speak candidly.
However, “all the big players want to avoid escalation, Iran wants to avoid this” because of threat of sanctions, the envoy insisted.
Absent that dire scenario, the impact on EU gas markets is likely to be limited, says Tom Marzec-Manser, head of gas analytics at commodities intelligence company ICIS — but more because of the last conflict than the most recent one.
“From a European gas pricing perspective, we’re still looking relatively OK and that’s been driven largely by weak demand. Many industrial consumers continue to use noticeably less gas than they did prior to the energy crisis last year, so consumption in Europe has remained low,” he said.
According to the European Commission, member states collectively shaved almost 20 percent from their natural gas use in the run-up to last winter, with industry slowing output and renewable power playing a much larger role in electricity generation. Despite that, consumption actually rose in October for the first time since the start of the war, in an early sign that businesses could be tentatively trying to restore lost productivity.
But even though the bloc’s gas reserves are more than 99 percent full ahead of schedule, prices have still remained stubbornly high across the Continent compared to other regions. That means Europeans are more at risk of short-term spikes in the cost of energy, with industry potentially having to slow down again if bills become unaffordable.
“We are in a much better situation than in 2022,” said Georg Zachmann, a senior fellow at the Bruegel energy think tank. “We have more heat pumps, power plants are back in the picture that we didn’t have available last year, and we’ve built more liquified natural gas terminals.” However, he warned, if member states lose focus on reducing demand and try to give their own industries a head start with subsidies, that could spark a wasteful race “that is essentially to everyone’s detriment.”
At the same time, winter in Europe isn’t what it used to be. Record-breaking temperatures have been recorded across the globe for the past four months, according to an EU Copernicus satellite monitoring report published this week, while last winter was the second-warmest ever recorded on the Continent. While that might be good news for conflict-prone fossil fuel supplies in the short term, it’s probably bad news for just about everything else in the not-so-much-longer term.
The EU’s energy crisis response is getting bigger, slowly. But so, too, is the threat posed by Russia’s freeze on Europe’s gas supply.
A new package of measures to bring down the price of gas and protect consumers this winter and beyond — including plans to fully leverage the EU’s collective buying power — will be formally proposed by the European Commission next week.
But there remains uncertainty about key aspects of the package — including whether the preferred intervention of many countries, an EU-wide cap on gas prices, will be part of it, and if so, in what form. It could also take until November to get next week’s proposals fully signed off and operational, officials said.
Even as energy ministers deliberated over the measures in Prague on Wednesday, Russia issued new, veiled warnings about the depths of Europe’s vulnerability.
Speaking at an energy conference in Moscow, the head of Gazprom Alexey Miller warned European homes could still freeze this winter even though EU countries have nearly filled their gas storage capacity.
At the same event, Vladimir Putin discussed the sabotage of the Nord Stream pipelines — an act that many Western governments suspect was the work of Russia. Then he added pointedly that the incident had shown how “any critical infrastructure in transport, energy or communication infrastructure is under threat — regardless of what part of the world it is located, by whom it is controlled, laid on the seabed or on land.”
Noting that one of the pipelines is still potentially operational after the attack, Putin insisted Russia was ready to send gas through it to ease Europe’s pain this winter — bringing his overarching strategy of gas blackmail against Europe right up to date.
“The ball, as they say, is on the side of the European Union. If they want it, let them just open the tap,” Putin said. “We are ready to supply additional volumes in the autumn-winter period.”
Putin may still be hoping that when the reality of winter without Russian gas begins to bite, European governments will be more open to such overtures — and more willing to rein in support for Ukraine in exchange for an energy lifeline.
For the EU’s part, Energy Commissioner Kadri Simson was clear that while the bloc faced “difficult times,” countries would withstand the challenges ahead if they “act together, decisively and in solidarity.”
Speaking at the close of an informal summit of EU energy ministers on Wednesday, she added that the next crisis package will also contain a proposal for a new benchmark price for gas and further measures to reduce demand across the bloc.
But while a row over capping the price of gas has dominated the debate in recent weeks, momentum has shifted to the idea of joint purchasing on the international market. It is hoped that through this measure the bloc can avoid the situation seen this year when member states outbid one another for supplies when filling gas storage facilities — driving up the price for all.
European Commissioner for Energy Kadri Simson | John Thys/AFP via Getty Images
In an informal policy paper issued on Wednesday, Germany and the Netherlands set how such a measure could work, by beefing up the existing EU Energy Platform, which was established months ago but then barely used. Efforts to buy gas jointly should be coupled with better EU-wide coordination of gas storage next year, the German and Dutch paper said.
The proposals point to the extent to which the EU is no longer simply planning how to survive this winter without rolling blackouts. It’s now firmly planning for a crisis next winter too.
Executive Director of the International Energy Agency Fatih Birol, who also attended Wednesday’s summit in Prague, warned ministers that “the next winter may well be even more difficult.”
That message was echoed in a sobering briefing from the EU Agency for the Cooperation of Energy Regulators, which outlined how challenging 2023 and potentially 2024 could be for the bloc’s energy supply. Amid an expected surge in demand in Asia for liquefied natural gas (LNG), the EU will face greater competition for limited LNG supplies from sources such as the U.S. and Qatar.
In short, every molecule of gas that remains in European storage after this winter might be vital — and Vladimir Putin knows it.
Victor Jack and America Hernandez provided additional reporting.
