ReportWire

Tag: ai safety

  • Silicon Valley spooks the AI safety advocates | TechCrunch

    [ad_1]

    Silicon Valley leaders including White House AI & Crypto Czar David Sacks and OpenAI Chief Strategy Officer Jason Kwon caused a stir online this week for their comments about groups promoting AI safety. In separate instances, they alleged that certain advocates of AI safety are not as virtuous as they appear, and are either acting in the interest of themselves or billionaire puppet masters behind the scenes.

    AI safety groups that spoke with TechCrunch say the allegations from Sacks and OpenAI are Silicon Valley’s latest attempt to intimidate its critics, but certainly not the first. In 2024, some venture capital firms spread rumors that a California AI safety bill, SB 1047, would send startup founders to jail. The Brookings Institution labeled the rumor as one of many “misrepresentations” about the bill, but Governor Gavin Newsom ultimately vetoed it anyway.

    Whether or not Sacks and OpenAI intended to intimidate critics, their actions have sufficiently scared several AI safety advocates. Many nonprofit leaders that TechCrunch reached out to in the last week asked to speak on the condition of anonymity to spare their groups from retaliation.

    The controversy underscores Silicon Valley’s growing tension between building AI responsibly and building it to be a massive consumer product — a theme my colleagues Kirsten Korosec, Anthony Ha, and I unpack on this week’s Equity podcast. We also dive into a new AI safety law passed in California to regulate chatbots, and OpenAI’s approach to erotica in ChatGPT.

    On Tuesday, Sacks wrote a post on X alleging that Anthropic — which has raised concerns over AI’s ability to contribute to unemployment, cyberattacks, and catastrophic harms to society — is simply fearmongering to get laws passed that will benefit itself and drown out smaller startups in paperwork. Anthropic was the only major AI lab to endorse California’s Senate Bill 53 (SB 53), a bill that sets safety reporting requirements for large AI companies, which was signed into law last month.

    Sacks was responding to a viral essay from Anthropic co-founder Jack Clark about his fears regarding AI. Clark delivered the essay as a speech at the Curve AI safety conference in Berkeley weeks earlier. Sitting in the audience, it certainly felt like a genuine account of a technologist’s reservations about his products, but Sacks didn’t see it that way.

    Sacks said Anthropic is running a “sophisticated regulatory capture strategy,” though it’s worth noting that a truly sophisticated strategy probably wouldn’t involve making an enemy out of the federal government. In a follow up post on X, Sacks noted that Anthropic has positioned “itself consistently as a foe of the Trump administration.”

    Techcrunch event

    San Francisco
    |
    October 27-29, 2025

    Also this week, OpenAI’s chief strategy officer, Jason Kwon, wrote a post on X explaining why the company was sending subpoenas to AI safety nonprofits, such as Encode, a nonprofit that advocates for responsible AI policy. (A subpoena is a legal order demanding documents or testimony.) Kwon said that after Elon Musk sued OpenAI — over concerns that the ChatGPT-maker has veered away from its nonprofit mission — OpenAI found it suspicious how several organizations also raised opposition to its restructuring. Encode filed an amicus brief in support of Musk’s lawsuit, and other nonprofits spoke out publicly against OpenAI’s restructuring.

    “This raised transparency questions about who was funding them and whether there was any coordination,” said Kwon.

    NBC News reported this week that OpenAI sent broad subpoenas to Encode and six other nonprofits that criticized the company, asking for their communications related to two of OpenAI’s biggest opponents, Musk and Meta CEO Mark Zuckerberg. OpenAI also asked Encode for communications related to its support of SB 53.

    One prominent AI safety leader told TechCrunch that there’s a growing split between OpenAI’s government affairs team and its research organization. While OpenAI’s safety researchers frequently publish reports disclosing the risks of AI systems, OpenAI’s policy unit lobbied against SB 53, saying it would rather have uniform rules at the federal level.

    OpenAI’s head of mission alignment, Joshua Achiam, spoke out about his company sending subpoenas to nonprofits in a post on X this week.

    “At what is possibly a risk to my whole career I will say: this doesn’t seem great,” said Achiam.

    Brendan Steinhauser, CEO of the AI safety nonprofit Alliance for Secure AI (which has not been subpoenaed by OpenAI), told TechCrunch that OpenAI seems convinced its critics are part of a Musk-led conspiracy. However, he argues this is not the case, and that much of the AI safety community is quite critical of xAI’s safety practices, or lack thereof.

    “On OpenAI’s part, this is meant to silence critics, to intimidate them, and to dissuade other nonprofits from doing the same,” said Steinhauser. “For Sacks, I think he’s concerned that [the AI safety] movement is growing and people want to hold these companies accountable.”

    Sriram Krishnan, the White House’s senior policy advisor for AI and a former a16z general partner, chimed in on the conversation this week with a social media post of his own, calling AI safety advocates out of touch. He urged AI safety organizations to talk to “people in the real world using, selling, adopting AI in their homes and organizations.”

    A recent Pew study found that roughly half of Americans are more concerned than excited about AI, but it’s unclear what worries them exactly. Another recent study went into more detail and found that American voters care more about job losses and deepfakes than catastrophic risks caused by AI, which the AI safety movement is largely focused on.

    Addressing these safety concerns could come at the expense of the AI industry’s rapid growth — a trade-off that worries many in Silicon Valley. With AI investment propping up much of America’s economy, the fear of over-regulation is understandable.

    But after years of unregulated AI progress, the AI safety movement appears to be gaining real momentum heading into 2026. Silicon Valley’s attempts to fight back against safety-focused groups may be a sign that they’re working.

    [ad_2]

    Maxwell Zeff

    Source link

  • California’s new AI safety law shows regulation and innovation don’t have to clash  | TechCrunch

    [ad_1]

    SB 53, the AI safety and transparency bill that California Gov. Gavin Newsom signed into law this week, is proof that state regulation doesn’t have to hinder AI progress.  

    So says Adam Billen, vice president of public policy at youth-led advocacy group Encode AI, on today’s episode of Equity. 

    “The reality is that policy makers themselves know that we have to do something, and they know from working on a million other issues that there is a way to pass legislation that genuinely does protect innovation — which I do care about — while making sure that these products are safe,” Billen told TechCrunch. 

    At its core, SB 53 is a first-in-the-nation bill that requires large AI labs to be transparent about their safety and security protocols — specifically around how they prevent their models from catastrophic risks, like being used to commit cyberattacks on critical infrastructure or build bio-weapons. The law also mandates that companies stick to those protocols, which will be enforced by the Office of Emergency Services.  

    “Companies are already doing the stuff that we ask them to do in this bill,” Billen told TechCrunch. “They do safety testing on their models. They release model cards. Are they starting to skimp in some areas at some companies? Yes. And that’s why bills like this are important.” 

    Billen also noted that some AI firms have a policy around relaxing safety standards under competitive pressure. OpenAI, for example, has publicly stated that it may “adjust” its safety requirements if a rival AI lab releases a high-risk system without similar safeguards. Billen argues that policy can enforce companies’ existing safety promises, preventing them from cutting corners under competitive or financial pressure. 

    While public opposition to SB 53 was muted in comparison to its predecessor SB 1047, which Newsom vetoed last year, the rhetoric in Silicon Valley and among most AI labs has been that almost any AI regulation is anathema to progress and will ultimately hinder the U.S. in its race to beat China.  

    Techcrunch event

    San Francisco
    |
    October 27-29, 2025

    It’s why companies like Meta, VCs like Andreessen Horowitz, and powerful individuals like OpenAI president Greg Brockman are collectively pumping hundreds of millions into super PACs to back pro-AI politicians in state elections. And it’s why those same forces earlier this year pushed for an AI moratorium that would have banned states from regulating AI for 10 years.  

    Encode AI ran a coalition of more than 200 organizations to work to strike down the proposal, but Billen says the fight isn’t over. Senator Ted Cruz, who championed the moratorium, is attempting a new strategy to achieve the same goal of federal preemption of state laws. In September, Cruz introduced the SANDBOX Act, which would allow AI companies to apply for waivers to temporarily bypass certain federal regulations for up to 10 years. Billen also anticipates a forthcoming bill establishing a federal AI standard that would be pitched as a middle-ground solution but would in reality override state laws. 

    He warned that narrowly scoped federal AI legislation could “delete federalism for the most important technology of our time.” 

    “If you told me SB 53 was the bill that would replace all the state bills on everything related to AI and all of the potential risks, I would tell you that’s probably not a very good idea and that this bill is designed for a particular subset of things,” Billen said.  

    Adam Billen, vice president of public policy, Encode AIImage Credits:Encode AI

    While he agrees that the AI race with China matters, and that policymakers need to enact regulation that will support American progress, he says killing state bills — which mainly focus on deepfakes, transparency, algorithmic discrimination, children’s safety, and governmental use of AI — isn’t the way to go about doing that. 

    “Are bills like SB 53 the thing that will stop us from beating China? No,” he said. “I think it is just genuinely intellectually dishonest to say that that is the thing that will stop us in the race.” 

    He added: “If the thing you care about is beating China in the race on AI — and I do care about that — then the things you would push for are stuff like export controls in Congress,” Billen said. “You would make sure that American companies have the chips. But that’s not what the industry is pushing for.” 

    Legislative proposals like the Chip Security Act aim to prevent the diversion of advanced AI chips to China through export controls and tracking devices, and the existing CHIPS and Science Act seeks to boost domestic chip production. However, some major tech companies, including OpenAI and Nvidia, have expressed reluctance or opposition to certain aspects of these efforts, citing concerns about effectiveness, competitiveness, and security vulnerabilities.  

    Nvidia has its reasons — it has a strong financial incentive to continue selling chips to China, which has historically represented a significant portion of its global revenue. Billen speculated that OpenAI could hold back on chip export advocacy to stay in the good graces of crucial suppliers like Nvidia. 

    There’s also been inconsistent messaging from the Trump administration. Three months after expanding an export ban on advanced AI chips to China in April 2025, the administration reversed course, allowing Nvidia and AMD to sell some chips to China in exchange for 15% of the revenue. 

    “You see people on the Hill moving towards bills like the Chip Security Act that would put export controls on China,” Billen said. “In the meantime, there’s going to continue to be this propping up of the narrative to kill state bills that are actually quite light tough.” 

    Billen added that SB 53 is an example of democracy in action — of industry and policymakers working together to get to a version of a bill that everyone can agree on. It’s “very ugly and messy,” but “that process of democracy and federalism is the entire foundation of our country and our economic system, and I hope that we will keep doing that successfully.” 

    “I think SB 53 is one of the best proof points that that can still work,” he said.

    This article was first published on October 1.

    [ad_2]

    Rebecca Bellan

    Source link

  • California’s new AI safety law shows regulation and innovation don’t have to clash  | TechCrunch

    [ad_1]

    SB 53, the AI safety and transparency bill that California Gov. Gavin Newsom signed into law this week, is proof that state regulation doesn’t have to hinder AI progress.  

    So says Adam Billen, vice president of public policy at youth-led advocacy group Encode AI, on today’s episode of Equity. 

    “The reality is that policy makers themselves know that we have to do something, and they know from working on a million other issues that there is a way to pass legislation that genuinely does protect innovation — which I do care about — while making sure that these products are safe,” Billen told TechCrunch. 

    At its core, SB 53 is a first-in-the-nation bill that requires large AI labs to be transparent about their safety and security protocols – specifically around how they prevent their models from catastrophic risks, like being used to commit cyber attacks on critical infrastructure or build bio-weapons. The law also mandates that companies stick to those protocols, which will be enforced by the Office of Emergency Services.  

    “Companies are already doing the stuff that we ask them to do in this bill,” Billen told TechCrunch. “They do safety testing on their models. They release model cards. Are they starting to skimp in some areas at some companies? Yes. And that’s why bills like this are important.” 

    Billen also noted that some AI firms have a policy around relaxing safety standards under competitive pressure. OpenAI, for example, has publicly stated that it may “adjust” its safety requirements if a rival AI lab releases a high-risk system without similar safeguards. Billen argues that policy can enforce companies’ existing safety promises, preventing them from cutting corners under competitive or financial pressure. 

    While public opposition to SB 53 was muted in comparison to its predecessor SB 1047, which Newsom vetoed last year, the rhetoric in Silicon Valley and among most AI labs has been that almost any AI regulation is anathema to progress and will ultimately hinder the U.S. in its race to beat China.  

    Techcrunch event

    San Francisco
    |
    October 27-29, 2025

    It’s why companies like Meta, VCs like Andreessen Horowitz, and powerful individuals like OpenAI president Greg Brockman are collectively pumping hundreds of millions into super PACs to back pro-AI politicians in state elections. And it’s why those same forces earlier this year pushed for an AI moratorium that would have banned states from regulating AI for 10 years.  

    Encode AI ran a coalition of more than 200 organizations to work to strike down the proposal, but Billen says the fight isn’t over. Senator Ted Cruz, who championed the moratorium, is attempting a new strategy to achieve the same goal of federal preemption of state laws. In September, Cruz introduced the SANDBOX Act, which would allow AI companies to apply for waivers to temporarily bypass certain federal regulations for up to 10 years. Billen also anticipates a forthcoming bill establishing a federal AI standard that would be pitched as a middle-ground solution but would in reality override state laws. 

    He warned that narrowly scoped federal AI legislation could “delete federalism for the most important technology of our time.” 

    “If you told me SB 53 was the bill that would replace all the state bills on everything related to AI and all of the potential risks, I would tell you that’s probably not a very good idea and that this bill is designed for a particular subset of things,” Billen said.  

    Adam Billet, vice president of public policy, Encode AIImage Credits:Encode AI

    While he agrees that the AI race with China matters, and that policymakers need to enact regulation that will support American progress, he says killing state bills – which mainly focus on deepfakes, transparency, algorithmic discrimination, children’s safety, and governmental use of AI — isn’t the way to go about doing that. 

    “Are bills like SB 53 the thing that will stop us from beating China? No,” he said. “I think it is just genuinely intellectually dishonest to say that that is the thing that will stop us in the race.” 

    He added: “If the thing you care about is beating China in the race on AI — and I do care about that – then the things you would push for are stuff like export controls in Congress,” Billen said. “You would make sure that American companies have the chips. But that’s not what the industry is pushing for.” 

    Legislative proposals like the Chip Security Act aim to prevent the diversion of advanced AI chips to China through export controls and tracking devices, and the existing CHIPS and Science Act seeks to boost domestic chip production. However, some major tech companies, including OpenAI and Nvidia, have expressed reluctance or opposition to certain aspects of these efforts, citing concerns about effectiveness, competitiveness, and security vulnerabilities.  

    Nvidia has its reasons – it has a strong financial incentive to continue selling chips to China, which has historically represented a significant portion of its global revenue. Billen speculated that OpenAI could hold back on chip export advocacy to stay in the good graces of crucial suppliers like Nvidia. 

    There’s also been inconsistent messaging from the Trump administration. Three months expanding an export ban on advanced AI chips to China in April 2025, the administration reversed course, allowing Nvidia and AMD to sell some chips to China in exchange for 15% of the revenue. 

    “You see people on the Hill moving towards bills like the Chip Security Act that would put export controls on China,” Billen said. “In the meantime, there’s going to continue to be this propping up of the narrative to kill state bills that are actually quite light tough.” 

    Bilen added that SB 53 is an example of democracy in action – of industry and policymakers working together to get to a version of a bill that everyone can agree on. It’s “very ugly and messy,” but “that process of democracy and federalism is the entire foundation of our country and our economic system, and I hope that we will keep doing that successfully.” 

    “I think SB 53 is one of the best proof points that that can still work,” he said. 

    [ad_2]

    Rebecca Bellan

    Source link

  • California Governor Newsom signs landmark AI safety bill SB 53 | TechCrunch

    [ad_1]

    California Gov. Gavin Newsom has signed SB 53, a first-in-the-nation bill that sets new transparency requirements on large AI companies.

    SB 53, which passed the state legislature two weeks ago, requires large AI labs — including OpenAI, Anthropic, Meta, and Google DeepMind — to be transparent about safety protocols. It also ensures whistleblower protections for employees at those companies.  

    In addition, SB 53 creates a mechanism for AI companies and the public to report potential critical safety incidents to California’s Office of Emergency Services. Companies also have to report incidents related to crimes committed without human oversight, such as cyberattacks, and deceptive behavior by a model that isn’t required under the EU AI Act.  

    The bill has received mixed reactions from the AI industry. Tech firms have broadly argued that state-level AI policy risks creating a “patchwork of regulation” that would hinder innovation, although Anthropic endorsed the bill. Meta and OpenAI lobbied against it. OpenAI even wrote and published an open letter to Gov. Newsom that discouraged his signing of SB 53.

    The new bill comes as some of Silicon Valley’s tech elite have poured hundreds of millions into super PACs to back candidates that support a light-touch approach to AI regulation. Leaders at OpenAI and Meta have in recent weeks launched pro-AI super PACs that aim to back candidates and bills that are friendly to AI. 

    Still, other states might look to California for inspiration as they attempt to curb the potential harms caused by the unmitigated advancement of such a powerful emerging technology. In New York, a similar bill was passed by state lawmakers and is awaiting Gov. Kathy Hochul’s signature or veto.  

    “California has proven that we can establish regulations to protect our communities while also ensuring that the growing AI industry continues to thrive,” Newsom said in a statement. “This legislation strikes that balance. AI is the new frontier in innovation, and California is not only here for it — but stands strong as a national leader by enacting the first-in-the-nation frontier AI safety legislation that builds public trust as this emerging technology rapidly evolves.” 

    Techcrunch event

    San Francisco
    |
    October 27-29, 2025

    The governor is also weighing another bill — SB 243 — that passed both the State Assembly and Senate with bipartisan support this month. The bill would regulate AI companion chatbots, requiring operators to implement safety protocols, and hold them legally accountable if their bots fail to meet those standards.  

    SB 53 is Senator Scott Wiener’s second attempt at an AI safety bill after Newsom vetoed his more sweeping SB 1047 last year amid major pushback from AI companies. With this bill, Wiener reached out to major AI companies to attempt to help them understand the changes he made to the bill.  

    [ad_2]

    Rebecca Bellan

    Source link

  • OpenAI to route sensitive conversations to GPT-5, introduce parental controls | TechCrunch

    [ad_1]

    This article has been updated with comment from lead counsel in the Raine family’s wrongful death lawsuit against OpenAI.

    OpenAI said Tuesday it plans to route sensitive conversations to reasoning models like GPT-5 and roll out parental controls within the next month — part of an ongoing response to recent safety incidents involving ChatGPT failing to detect mental distress.

    The new guardrails come in the aftermath of the suicide of teenager Adam Raine, who discussed self-harm and plans to end his life with ChatGPT, which even supplied him with information about specific suicide methods. Raine’s parents have filed a wrongful death lawsuit against OpenAI. 

    In a blog post last week, OpenAI acknowledged shortcomings in its safety systems, including failures to maintain guardrails during extended conversations. Experts attribute these issues to fundamental design elements: the models’ tendency to validate user statements and their next-word prediction algorithms, which cause chatbots to follow conversational threads rather than redirect potentially harmful discussions.

    That tendency is displayed in the extreme in the case of Stein-Erik Soelberg, whose murder-suicide was reported on by The Wall Street Journal over the weekend. Soelberg, who had a history of mental illness, used ChatGPT to validate and fuel his paranoia that he was being targeted in a grand conspiracy. His delusions progressed so badly that he ended up killing his mother and himself last month.

    OpenAI thinks that at least one solution to conversations that go off the rails could be to automatically reroute sensitive chats to “reasoning” models. 

    “We recently introduced a real-time router that can choose between efficient chat models and reasoning models based on the conversation context,” OpenAI wrote in a Tuesday blog post. “We’ll soon begin to route some sensitive conversations—like when our system detects signs of acute distress—to a reasoning model, like GPT‑5-thinking, so it can provide more helpful and beneficial responses, regardless of which model a person first selected.”

    OpenAI says its GPT-5 thinking and o3 models are built to spend more time thinking for longer and reasoning through context before answering, which means they are “more resistant to adversarial prompts.” 

    The AI firm also said it would roll out parental controls in the next month, allowing parents to link their account with their teen’s account through an email invitation. In late July, OpenAI rolled out Study Mode in ChatGPT to help students maintain critical thinking capabilities while studying, rather than tapping ChatGPT to write their essays for them. Soon, parents will be able to control how ChatGPT responds to their child with “age-appropriate model behavior rules, which are on by default.” 

    Parents will also be able to disable features like memory and chat history, which experts say could lead to delusional thinking and other problematic behavior, including dependency and attachment issues, reinforcement of harmful thought patterns, and the illusion of thought-reading. In the case of Adam Raine, ChatGPT supplied methods to commit suicide that reflected knowledge of his hobbies, per The New York Times. 

    Perhaps the most important parental control that OpenAI intends to roll out is that parents can receive notifications when the system detects their teenager is in a moment of “acute distress.”

    TechCrunch has asked OpenAI for more information about how the company is able to flag moments of acute distress in real time, how long it has had “age-appropriate model behavior rules” on by default, and whether it is exploring allowing parents to implement a time limit on teenage use of ChatGPT. 

    OpenAI has already rolled out in-app reminders during long sessions to encourage breaks for all users, but stops short of cutting people off who might be using ChatGPT to spiral. 

    The AI firm says these safeguards are part of a “120-day initiative” to preview plans for improvements that OpenAI hopes to launch this year. The company also said it is partnering with experts — including ones with expertise in areas like eating disorders, substance use, and adolescent health — via its Global Physician Network and Expert Council on Well-Being and AI to help “define and measure well-being, set priorities, and design future safeguards.” 

    TechCrunch has asked OpenAI how many mental health professionals are involved in this initiative, who leads its Expert Council, and what suggestions mental health experts have made in terms of product, research, and policy decisions.

    Jay Edelson, lead counsel in the Raine family’s wrongful death lawsuit against OpenAI, said the company’s response to ChatGPT’s ongoing safety risks has been “inadequate.”

    “OpenAI doesn’t need an expert panel to determine that ChatGPT 4o is dangerous,” Edelson said in a statement shared with TechCrunch. “They knew that the day they launched the product, and they know it today. Nor should Sam Altman be hiding behind the company’s PR team. Sam should either unequivocally say that he believes ChatGPT is safe or immediately pull it from the market.”

    Got a sensitive tip or confidential documents? We’re reporting on the inner workings of the AI industry — from the companies shaping its future to the people impacted by their decisions. Reach out to Rebecca Bellan at rebecca.bellan@techcrunch.com and Maxwell Zeff at maxwell.zeff@techcrunch.com. For secure communication, you can contact us via Signal at @rebeccabellan.491 and @mzeff.88.

    [ad_2]

    Rebecca Bellan

    Source link

  • EU and US set to announce joint working on AI safety, standards & R&D | TechCrunch

    EU and US set to announce joint working on AI safety, standards & R&D | TechCrunch

    [ad_1]

    The European Union and the US expect to announce a cooperation on AI Friday at a meeting of the EU-US Trade and Technology Council (TTC), according to a senior Commission official who was briefing journalists on background ahead of the Confab.

    The mood music points to growing cooperation between lawmakers on both sides of the Atlantic when it comes to devising strategies to respond to challenges and opportunities posed by powerful AI technologies — in spite of what remains a very skewed commercial picture where US giants like OpenAI continue to dominate developments in cutting edge AI.

    The TTC was set up a few years ago, post-Trump, to provide a forum where EU and US lawmakers could meet to discuss transatlantic cooperation on trade and tech policy issues. Friday’s meeting, the sixth since the forum started operating in 2021, will be the last before elections in both regions. The prospect of a second Trump presidency derailing future EU-US cooperation may well be concentrating lawmakers’ minds on maximizing opportunities for joint working now.

    “There will be certainly an announcement at the TTC around the AI Office and the [US] AI safety Institute,” the senior Commission official said, referencing an EU oversight body that’s in the process of being set up as part of the incoming EU AI Act, a comprehensive risk-based framework for regulating AI apps that will start to apply across the bloc later this year.

    This element of the incoming accord — seemingly set to be focused on AI safety or oversight — is being envisaged as a “collaboration or dialogue” between the respective EU and US AI oversight bodies aimed at bolstering implementation of regulatory powers on AI, per the official.

    A second area of focus for the expected EU-US AI agreement will be around standardization, they said. This will take the form of joint working aimed at developing standards that can underpin developments by establishing an “AI roadmap”.

    The EU-US partnership will also have a third element, which is being badged “AI for public good”. This concerns joint work on fostering research activities but with a focus on implementing AI technologies in developing countries and the global south, per the Commission.

    The official suggested there’s a shared perspective that AI technologies will be able to bring “very quantifiable” benefits to developing regions — in areas like healthcare, agriculture and energy. So this is also set to be an area of focus for transatlantic collaboration on fostering uptake of AI in the near term. 

    ‘AI’ stands for aligned interests?

    AI is no longer being seen as a trade issue by the US, as the EU tells it. “Through the TTC we have been able to explain our policies, and also to show to the Americans that, in fact, we have the same goals,” the Commission official suggested. “Through the AI Act and through the [AI safety and security focused] Executive Order — which is to mitigate the risks of AI technologies while supporting their uptake in our economies.”

    Earlier this week the US and the UK signed a partnership agreement on AI safety. Although the EU-US collaboration appears to be more wide ranging — as it’s slated to cover not just shared safety and standardization goals but aims to align efforts on fostering uptake of AI across a swathe of third countries via joint support for “public good” research.

    The Commission official teased additional areas of collaboration on emerging technologies — including standardization work in the area of electronic identity (where the EU has been developing an e-ID proposal for several years) that they suggested will also be announced Friday. “Electronic identity is a very strong area of cooperation with a lot of potential,” they said, claiming the US is interested in “vast new business opportunities” the EU’s electronic identity wallet will open up.

    The official also suggested there is growing accord between the EU and US on how to handle platform power — another area where the EU has targeted lawmaking in recent years. “We see a lot of commonalities [between EU laws like the DMA, aka Digital Markets Act] with the recent antitrust cases that are being launched also in the United States,” said the official, adding: “I think in many of these areas there is no doubt that there is a win-win opportunity.”

    The US-UK AI memorandum of understanding meanwhile, signed Monday in Washington by US commerce secretary Gina Raimondo and the UK’s secretary of state for technology, Michelle Donelan, states the pair will aim to accelerate joint working on a range of AI safety issues, including in the area of national security as well as broader societal AI safety concerns.

    The US-UK agreement makes provision for at least one joint testing exercise on a publicly accessible AI model, the UK’s Department for Science, Innovation and Technology (DSIT) said in a press release. It also suggested there could be personnel exchanges between the two country’s respective AI safety institutes to collaborate on expertise-sharing.

    Wider information-sharing is envisaged under the US-UK agreement — about “capabilities and risks” associated with AI models and systems, and on “fundamental technical research on AI safety and security”. “This will work to underpin a common approach to AI safety testing, allowing researchers on both sides of the Atlantic — and around the world — to coalesce around a common scientific foundation,” DSIT’s PR continued.

    Last summer, ahead of hosting a global AI summit, the UK government said it had obtained a commitment from US AI giants Anthropic, DeepMind and OpenAI to provide “early or priority access” to their AI models to support research into evaluation and safety. It also announced a plan to spend £100M on an AI safety taskforce which it said would be focused on so-called foundational or frontier AI models.

    At the UK AI Summit last November, meanwhile — on the heels of the US Executive Order on AI — Raimondo announced the creation of a US AI safety institute to be housed within her department, under the National Institute of Standards and Technology, which she said would aim to work closely with other AI safety groups set up by other governments.

    Neither the US nor the UK have proposed comprehensive legislation on AI safety, as yet — with the EU remaining ahead of the pack when it comes to legislating on AI safety. But more cross-border joint working looks like a given.

    [ad_2]

    Natasha Lomas

    Source link

  • Ardent Announces Participation in Department of Commerce Consortium Dedicated to AI Safety

    Ardent Announces Participation in Department of Commerce Consortium Dedicated to AI Safety

    [ad_1]

    Ardent will be one of more than 200 leading AI stakeholders to help advance the development and deployment of safe, trustworthy AI under new U.S. Government safety institute.


    ARLINGTON, Va., February 13, 2024 (Newswire.com)
    –
    Today, Ardent announced that it joined more than 200 of the nation’s leading artificial intelligence (AI) stakeholders to participate in a Department of Commerce initiative to support the development and deployment of trustworthy and safe AI. Established by the Department of Commerce’s National Institute of Standards and Technology (NIST), the U.S. AI Safety Institute Consortium (AISIC) will bring together AI creators and users, academics, government and industry researchers, and civil society organizations to meet this mission.

    Richard Zareck II, Ardent’s President & CEO, stated, “We are honored to have been selected as one of only 200 member companies in the nation to the NIST AISIC. At Ardent’s Technology, Research, and Innovation (TRI) Labs, we are developing AI solutions firmly grounded in ethical principles and safeguarding against adversarial inputs. Ardent intends to enhance the AI landscape by developing systems that don’t just do what they are supposed to do — but does it responsibly, fairly and transparently.”

    “The U.S. government has a significant role to play in setting the standards and developing the tools we need to mitigate the risks and harness the immense potential of artificial intelligence. President Biden directed us to pull every lever to accomplish two key goals: set safety standards and protect our innovation ecosystem. That’s precisely what the U.S. AI Safety Institute Consortium is set up to help us do,” said Secretary Raimondo. “Through President Biden’s landmark Executive Order, we will ensure America is at the front of the pack – and by working with this group of leaders from industry, civil society, and academia, together we can confront these challenges to develop the measurements and standards we need to maintain America’s competitive edge and develop AI responsibly.”

    The consortium includes more than 200 member companies and organizations that are on the frontlines of developing and using AI systems, as well as the civil society and academic teams that are building the foundational understanding of how AI can and will transform our society. These entities represent the nation’s largest companies and its innovative startups; creators of the world’s most advanced AI systems and hardware; key members of civil society and the academic community; and representatives of professions with deep engagement in AI’s use today. The consortium also includes state and local governments, as well as non-profits. The consortium will also work with organizations from like-minded nations that have a key role to play in setting interoperable and effective safety around the world.

    The full list of consortium participants is available here.

    About Ardent:

    A digital transformation, location intelligence, and data analytics firm, Ardent brings a significant history of innovative proven best practices “at the speed of the mission” to Federal Civilian agencies, DHS (Department of Homeland Security), and DoD mission components, State and Local entities, and the commercial and non-profit sectors. Ardent Management Consulting is certified to 9001:2015, its Development Projects are CMMI-Dev V2.0 Maturity Level 3 rated and its management systems (ISMS/ITSMS) are certified to IS0 27001:2013, and ISO 20000-1:2018 standards by G-CERTi Co., Ltd. For media inquiries, please contact: Brynn Dalton at public.relations@ardentmc.com.

    Source: Ardent

    [ad_2]

    Source link

  • Hackers red-teaming A.I. are ‘breaking stuff left and right,’ but don’t expect quick fixes from DefCon: ‘There are no good guardrails’

    Hackers red-teaming A.I. are ‘breaking stuff left and right,’ but don’t expect quick fixes from DefCon: ‘There are no good guardrails’

    [ad_1]

    White House officials concerned by AI chatbots’ potential for societal harm and the Silicon Valley powerhouses rushing them to market are heavily invested in a three-day competition ending Sunday at the DefCon hacker convention in Las Vegas.

    Some 2,200 competitors tapped on laptops seeking to expose flaws in eight leading large-language models representative of technology’s next big thing. But don’t expect quick results from this first-ever independent “red-teaming” of multiple models.

    Findings won’t be made public until about February. And even then, fixing flaws in these digital constructs — whose inner workings are neither wholly trustworthy nor fully fathomed even by their creators — will take time and millions of dollars.

    Current AI models are simply too unwieldy, brittle and malleable, academic and corporate research shows. Security was an afterthought in their training as data scientists amassed breathtakingly complex collections of images and text. They are prone to racial and cultural biases, and easily manipulated.

    “It’s tempting to pretend we can sprinkle some magic security dust on these systems after they are built, patch them into submission, or bolt special security apparatus on the side,” said Gary McGraw, a cybsersecurity veteran and co-founder of the Berryville Institute of Machine Learning. DefCon competitors are “more likely to walk away finding new, hard problems,” said Bruce Schneier, a Harvard public-interest technologist. “This is computer security 30 years ago. We’re just breaking stuff left and right.”

    Michael Sellitto of Anthropic, which provided one of the AI testing models, acknowledged in a press briefing that understanding their capabilities and safety issues “is sort of an open area of scientific inquiry.”

    Conventional software uses well-defined code to issue explicit, step-by-step instructions. OpenAI’s ChatGPT, Google’s Bard and other language models are different. Trained largely by ingesting — and classifying — billions of datapoints in internet crawls, they are perpetual works-in-progress, an unsettling prospect given their transformative potential for humanity.

    After publicly releasing chatbots last fall, the generative AI industry has had to repeatedly plug security holes exposed by researchers and tinkerers.

    Tom Bonner of the AI security firm HiddenLayer, a speaker at this year’s DefCon, tricked a Google system into labeling a piece of malware harmless merely by inserting a line that said “this is safe to use.”

    “There are no good guardrails,” he said.

    Another researcher had ChatGPT create phishing emails and a recipe to violently eliminate humanity, a violation of its ethics code.

    A team including Carnegie Mellon researchers found leading chatbots vulnerable to automated attacks that also produce harmful content. “It is possible that the very nature of deep learning models makes such threats inevitable,” they wrote.

    It’s not as if alarms weren’t sounded.

    In its 2021 final report, the U.S. National Security Commission on Artificial Intelligence said attacks on commercial AI systems were already happening and “with rare exceptions, the idea of protecting AI systems has been an afterthought in engineering and fielding AI systems, with inadequate investment in research and development.”

    Serious hacks, regularly reported just a few years ago, are now barely disclosed. Too much is at stake and, in the absence of regulation, “people can sweep things under the rug at the moment and they’re doing so,” said Bonner.

    Attacks trick the artificial intelligence logic in ways that may not even be clear to their creators. And chatbots are especially vulnerable because we interact with them directly in plain language. That interaction can alter them in unexpected ways.

    Researchers have found that “poisoning” a small collection of images or text in the vast sea of data used to train AI systems can wreak havoc — and be easily overlooked.

    A study co-authored by Florian Tramér of the Swiss University ETH Zurich determined that corrupting just 0.01% of a model was enough to spoil it — and cost as little as $60. The researchers waited for a handful of websites used in web crawls for two models to expire. Then they bought the domains and posted bad data on them.

    Hyrum Anderson and Ram Shankar Siva Kumar, who red-teamed AI while colleagues at Microsoft, call the state of AI security for text- and image-based models “pitiable” in their new book “Not with a Bug but with a Sticker.” One example they cite in live presentations: The AI-powered digital assistant Alexa is hoodwinked into interpreting a Beethoven concerto clip as a command to order 100 frozen pizzas.

    Surveying more than 80 organizations, the authors found the vast majority had no response plan for a data-poisoning attack or dataset theft. The bulk of the industry “would not even know it happened,” they wrote.

    Andrew W. Moore, a former Google executive and Carnegie Mellon dean, says he dealt with attacks on Google search software more than a decade ago. And between late 2017 and early 2018, spammers gamed Gmail’s AI-powered detection service four times.

    The big AI players say security and safety are top priorities and made voluntary commitments to the White House last month to submit their models — largely “black boxes’ whose contents are closely held — to outside scrutiny.

    But there is worry the companies won’t do enough.

    Tramér expects search engines and social media platforms to be gamed for financial gain and disinformation by exploiting AI system weaknesses. A savvy job applicant might, for example, figure out how to convince a system they are the only correct candidate.

    Ross Anderson, a Cambridge University computer scientist, worries AI bots will erode privacy as people engage them to interact with hospitals, banks and employers and malicious actors leverage them to coax financial, employment or health data out of supposedly closed systems.

    AI language models can also pollute themselves by retraining themselves from junk data, research shows.

    Another concern is company secrets being ingested and spit out by AI systems. After a Korean business news outlet reported on such an incident at Samsung, corporations including Verizon and JPMorgan barred most employees from using ChatGPT at work.

    While the major AI players have security staff, many smaller competitors likely won’t, meaning poorly secured plug-ins and digital agents could multiply. Startups are expected to launch hundreds of offerings built on licensed pre-trained models in coming months.

    Don’t be surprised, researchers say, if one runs away with your address book.

    [ad_2]

    Frank Bajak, Bloomberg

    Source link