Logo
FrontierNews.ai

A New Bet on AI Safety: How Researchers Are Getting Funded to Build Guardrails Before AGI

A new grant program is channeling funding directly into AI safety research, offering researchers and teams up to $100,000 to develop projects that reduce existential risks from advanced AI systems. The initiative, open to individuals, teams, and both nonprofit and for-profit organizations, represents a significant institutional commitment to addressing AI safety concerns before artificial general intelligence (AGI), a hypothetical AI system that could match or exceed human intelligence across all domains, becomes a near-term reality.

What Types of AI Safety Projects Are Getting Funded?

The program distributes support across eight distinct focus areas, each designed to tackle different dimensions of AI risk and safety. Rather than funding a single narrow approach, the initiative recognizes that AI safety requires work across multiple technical and governance domains.

  • AI for Security: Projects that use AI systems to identify vulnerabilities and strengthen defenses against malicious actors.
  • Decentralized and Cooperative AI: Research into AI systems designed to work collaboratively while maintaining safety properties and reducing concentration of power.
  • Longevity Biotechnology: AI applications aimed at extending human healthspan, which some researchers argue could reduce existential pressures.
  • Molecular Nanotechnology: Exploring how AI can advance molecular-scale engineering while managing associated risks.
  • Brain-Computer Interfaces: Developing safe integration pathways between human cognition and AI systems.

The program prioritizes projects based on four key criteria: alignment with these focus areas, feasibility within short AGI timelines, preference for open-source development, and potential to reduce existential risks. This framework suggests the funders believe AGI could arrive sooner than many mainstream technology companies assume, making near-term safety work urgent.

How Can Researchers Access This Funding and Support?

Beyond the financial grants, which range from $10,000 to $100,000, the program offers a comprehensive ecosystem designed to accelerate collaboration and reduce barriers to entry for safety researchers. This multi-layered support model reflects a recognition that funding alone isn't sufficient; researchers need infrastructure, community, and access to computational resources.

  • Physical Hub Access: Grant recipients gain zero-cost office space at designated hubs in Berlin and San Francisco, creating in-person research communities.
  • Compute Resources: Local, private computing infrastructure is provided at no cost, removing a major financial barrier for teams developing and testing AI safety techniques.
  • Collaborative Events: Recipients are invited to travel-paid workshops and collaborative sprints designed to advance their specific research areas and foster peer learning.
  • Community Integration: The program strongly prioritizes "AI-first" applicants who commit to being active, in-person members at one of the hubs, with funding-only applications considered only in exceptional circumstances.

This emphasis on in-person collaboration suggests the funders believe that breakthrough safety research requires sustained interaction between researchers, not just remote coordination. The requirement that most applicants participate in hub communities also creates a selection mechanism favoring researchers who are deeply committed to the work.

Why Is AGI Timeline Feasibility a Funding Priority?

The program's explicit focus on "feasibility within short artificial general intelligence timelines" signals a shift in how some institutions are thinking about AI risk. Rather than treating AGI as a distant, speculative concern, this funding approach assumes AGI could arrive within years rather than decades, making safety work urgent and time-sensitive.

This timeline assumption has practical implications for which projects get prioritized. Research that requires five to ten years to mature may be less attractive than work that can produce safety insights or tools within one to three years. The emphasis on open-source development also reflects a belief that safety benefits should be widely available, not locked behind proprietary systems, as AGI development accelerates across multiple organizations and countries.

The funding initiative represents a notable institutional response to growing concerns about AI safety and existential risk. By combining financial support with infrastructure, compute resources, and community, the program aims to attract and retain top talent in a field that has historically struggled to compete with well-funded AI development teams. Whether this approach will meaningfully accelerate progress on AI safety remains to be seen, but the scale and structure of the initiative suggest that major funders are taking existential AI risk seriously enough to invest substantially in preventive research.