10:T415c,## **Project Summary** This project builds upon [our successful Manifund 2023-supported research](https://manifund.org/projects/scoping-developmental-interpretability-xg55b33wsfc) that validated [Developmental Interpretability](https://www.lesswrong.com/posts/TjaeCWvLZtEDAS5Ex/towards-developmental-interpretability) (DevInterp) as an approach to understanding the internal structure of neural networks. We ([Timaeus](https://timaeus.co/)) are seeking funding to continue this research and extend our runway from 6 months to \~1 year. In particular, this funding would go to a series of projects (described below) that apply our existing research to address immediate, real-world safety concerns. This has the ultimate goal of leading to novel _understanding-based_ evals. ## **Developmental Interpretability & Singular Learning Theory** We [initially](https://manifund.org/projects/scoping-developmental-interpretability-xg55b33wsfc) set out to establish the viability of [Developmental Interpretability](https://www.lesswrong.com/posts/TjaeCWvLZtEDAS5Ex/towards-developmental-interpretability), an application of Singular Learning Theory (SLT) to interpreting the formation of structure in neural networks. [This succeeded](https://manifund.org//projects/scoping-developmental-interpretability-xg55b33wsfc?tab=comments#4882aabb-38b2-4d6c-a1a0-a57f43a96d57). Together with our collaborators, * **We demonstrated that SLT can inform the development of new measurements**, such as local learning coefficient (LLC) estimation. This has now been successfully applied to models with up to 100s of millions of parameters \[[Lau et al. 2023](https://arxiv.org/abs/2308.12108); [Furman & Lau 2024](https://arxiv.org/abs/2402.03698)\]. * **We confirmed that phase transitions are real and our tools can detect them**, in simple models like the toy model of superposition and deep linear networks \[[Chen et al. 2023](https://arxiv.org/abs/2310.06301); [Furman & Lau 2024](https://arxiv.org/abs/2402.03698)\]. * **We showed that hidden stage-wise development is common** in larger more realistic models. While larger models do not undergo the same kind of “sharp” transitions as smaller models, the same theory applies to developmental stages. We showed that these stages can be revealed using the LLC \[[Hoogland et al. 2024](https://arxiv.org/abs/2402.02364)\]. In some stages the loss changes only slightly while the LLC decreases substantially, reflecting qualitative changes in model computation. The upshot is that the LLC is a scalable and theoretically well-justified measure of model complexity, which can be used to study the development of structure in neural networks. We are now confident that DevInterp is a viable research agenda and expect that LLC estimation will become widely used in AI safety. ## **Near-Term Applications** We will continue to work towards gaining an understanding of the fundamental units of computation in neural networks and how they develop (e.g. “circuits”) but we are also excited to take the understanding and tools that we have developed so far and apply them immediately to practical problems in AI safety. It is these practical applications that we are requesting funding for. **We believe the LLC is now ready to be applied to models with billions of parameters, to radically improve our ability to reason qualitatively and quantitatively** about questions like: * How deep is safety training, really? * How does safety fine-tuning / RLHF change model computation? * How does additional fine-tuning affect safety training? The answers to these questions hinge on _generalization_: how do pre-trained models generalize from the (relatively) small number of examples in the safety fine-tuning / RLHF training data? When more, or different, examples are provided, how does that generalization change? These are questions that SLT and DevInterp are well-positioned to answer. In the first case because SLT is a theory of generalization and the LLC is how we quantify the effect of loss landscape geometry on generalization, and in the second case because _studying how models change over training_ (DevInterp) is very similar to _studying how models change over fine-tuning_. **We are excited about working on these questions for three reasons**: * They play to the comparative advantages of our research agenda. * They prepare us for studying [sharp left turns](https://www.lesswrong.com/posts/GNhMPAWcfBCASy8e6/a-central-ai-alignment-problem-capabilities-generalization) which we view as a central problem in AI alignment. * They are on the critical path to building [understanding-based evals](https://www.alignmentforum.org/posts/uqAdqrvxqGqeBHjTP/towards-understanding-based-safety-evaluations), a core goal for Timaeus. ### **Project(s) Description** The near-term applications of our research to AI safety are based on the local learning coefficient (LLC). The LLC is a measure of how large a perturbation in weights is required to meaningfully change model behavior. In forthcoming work \[Wang et al.\] described below, we show that restricted versions of the LLC can quantify how large a perturbation in _specific_ weights is required to meaningfully change model behavior _on a particular dataset_ (e.g. to measure how much a particular attention head is specialized to code, or how easily a given layer can recover examples of illicit behaviors that have been “trained out” by safety fine-tuning). These restricted LLCs are useful in studying how components of the model differentiate and mature over training, and in tracking which parts of the model have information about which kinds of datasets. We emphasize that in our experience the LLC and its restricted variants _often reveal information that is not obviously accessible through the loss_ or other metrics alone. This empirical fact, combined with its fundamental status in the mathematical theory of Bayesian statistics, justify our focus on the LLC as a tool of choice when trying to understand models. The particular projects we have in mind are: * **Project: Simplicity biases and deceptive alignment.** In their recent “sleeper agents” paper [Hubinger et al.](https://arxiv.org/abs/2401.05566) show that adversarial training fails to remove a backdoor behavior trained into an LLM, and instead causes it to learn a more specific backdoor. They point out that this is highly relevant to safety training, since “once a model develops a harmful or unintended behavior, training on examples where the model exhibits the harmful behavior might serve only to hide the behavior rather than remove it entirely”. They conjecture an explanation in terms of a bias of SGD towards _simpler modifications._ This phenomenon and the conjectured bias can be understood in terms of the free energy formula within SLT, and we will investigate the use of restricted LLCs for studying this phenomenon. * **Project: Backdoor detection with data restricted LLCs.** If a model is backdoored it may “compute differently” on inputs from slightly different distributions, in a way that is unanticipated. Different input distributions determine different population loss landscape geometries, and the data restricted LLC reflects these changes; consequently, we expect the data-restricted LLC to be sensitive to changes in model computation as a result of small changes in input distribution. We will investigate how this can be used to detect backdoors. * **Project: Understanding algorithm change in many-shot jailbreaking.** As long-context LLMs become available, the study of [many-shot in-context learning](https://arxiv.org/abs/2404.11018) has become increasingly critical to AI safety. As a follow-up to our ongoing work using SLT and DevInterp to study the fine-tuning process (described below) we will study [many-shot jailbreaking](https://www.anthropic.com/research/many-shot-jailbreaking). In the forthcoming Carroll et al., we study how the algorithms learned by transformers change over training, according to the free energy formula in SLT, and we have ideas on how to adapt this to study how the mode of computation varies over long contexts. * **Project: A minimal understanding-based eval.** By scaling data and weight restricted LLCs to large open-weight models, combined with the above projects using these tools to reason about the effect of safety fine-tuning and adversarial training, we will produce an eval that goes beyond behavioral evaluation and demonstrate its application to simple examples of engineered deceptive alignment and backdoors. A subset of these projects involve a close collaboration and partnership with the [Gradient Institute](https://www.gradientinstitute.org/). The Gradient Institute is an independent nonprofit research institute based in Sydney, Australia, which over the past five years has become a trusted advisor to corporations and government in Australia on AI policy development.This partnership combines our technical expertise with Gradient's policy experience, ensuring that our research findings have a clear path to inform and influence AI policy and regulation. ## **Research Details** To provide more context on our research, two projects that will be completed this month: * **G. Wang, J. Hoogland, S. van Wingerden, Z. Furman, D. Murfet “Differentiation and Specialization in Language Models via the Restricted Local Learning Coefficient”** introduces the weight and data-restricted LLCs and shows that (a) attention heads in a 3M parameter transformer differentiate over training in ways that are tracked by the weight-restricted LLC, (b) some induction heads are partly specialized to code, and this is reflected in the data-restricted LLC on code-related tasks, (c) all attention heads follow the same pattern in which their data-restricted LLCs first increase then decrease, which appears similar to the critical periods studied by [Achille-Rovere-Soatto](https://arxiv.org/abs/1711.08856). * **L. Carroll, J. Hoogland, D. Murfet “Retreat from Ridge: Studying Algorithm Choice in Transformers using Essential Dynamics”** studies the retreat from ridge phenomena following [Raventós et al.](https://arxiv.org/abs/2306.15063) and resolves the mystery of apparent non-Bayesianism there, by showing that over training for an in-context linear regression problem there is tradeoff between in-context ridge regression (a simple but high error solution) and another solution more specific to the dataset (which is more complex but lower error). This gives a striking example of the “accuracy vs. simplicity” tradeoff made quantitative by the free energy formula in SLT. A project recently started with the Gradient Institute, which we expect to develop in parallel with the work on deceptive alignment: * **Project: Measuring reversibility of safety fine-tuning:** We can see ways to use restricted LLC estimation to quantify how “deep”/reversibile safety fine-tuning is in open-weights models at the scale of 7B parameters in a similar vein to the recent work of [Peng et al.](https://arxiv.org/abs/2405.17374v2) on the “safety landscape”. The success of the new proposed projects will depend on improvements in the underlying technology of LLC estimation, which Timaeus will continue to push forward. We expect several of these projects to involve open-weights models on the scale of billions of parameters. At present we have extensive experience running LLC estimation on models with millions of parameters and more limited experience with models at the scale of hundreds of millions of parameters (GPT2-small). We do not foresee fundamental obstacles in scaling another order of magnitude, but at each scale we need to invest researcher time in developing best practices for hyperparameter selection, baseline sanity checks, etc. ## **How will this funding be used?** This funding will be used to support the research projects listed above, mainly by paying for researchers and compute. We are seeking $670k in additional funding to complete the outlined parts of research agenda, to be spent over the course of 8 months: * $100,000 to complete partially funded projects. * $410,000 to complete unfunded projects. * $160,000 for accelerating our research timelines (2 additional RAs and increased compute) ### **Funding Breakdown** * 355k: RAs + Researchers (3.4 FTE, 8 months) * 30k: Travel Support & Research Retreats * 145k: Core Team + Support Staff (1.8 FTE, 8 months) * 55k: Compute * 85k: Fiscal Sponsorship Costs & Buffer ### **Timeline** By default, we are targeting publication in the major conferences and journals, subject to case-by-case considerations of capability risk. Our timeline for submissions (the actual conferences take place later) is: * **ICML** (January/February 2025, Partially funded) * Project: Measuring reversibility of safety-fine tuning * Project: Understanding algorithm change in many-shot jailbreaking * **NeurIPS** (May 2025, not yet funded) * Project: Simplicity biases and deceptive alignment * Project: Backdoor detection with data restricted LLCs * **ICLR** (September 2025, aiming for completion in August 2025, not yet funded) * Project: A minimal understanding-based eval ### **Additional Considerations** * **Potential Harm:** While our research aims to improve AI safety, it could potentially be misused to accelerate capabilities development. We are committed to conducting risk assessments on a project-by-project basis and working with the community to mitigate any potential harm. * **Other Funding:** Since October 2023, Timaeus has secured $800k in funding from Manifund, SFF, LTFF, and AISTOF, providing runway until January 1st 2025\. The Gradient Institute is jointly funding the projects listed under our partnership. We are actively seeking additional funding to support our ongoing and future research, including at SFF and Foresight. ## **Track Record** [Timaeus](https://timaeus.co/) is currently a [team of 6.5 FTE](https://timaeus.co/team), with a strong track record in advancing DevInterp and SLT: * **Research:** We have successfully proposed and validated a new research direction within AI safety through a combination of careful empirical and theoretical work and on an accelerated time frame. See the references linked at the end. * **Expertise:** We are working in close collaboration with [Daniel Murfet](http://therisingsea.org/cv.pdf)’s group at the University of Melbourne and with other top researchers at several universities and AI labs. Daniel Murfet is responsible for originating the developmental interpretability agenda and is senior author on many of the papers we are writing. * **Outreach:** We have actively engaged with the AI alignment community through conferences, workshops, talks, mentoring (via MATS, Athena, ERA), and other forms of outreach. This has led to widespread interest and [endorsements](https://docs.google.com/document/d/1u9as2E9S3LO1R%5FkE-SE-Q7OBYvKgz3gHiB7Pa9wqaZ4/edit) by many prominent researchers in the field, including John Wentworth, Daniel Filan, and Vanessa Kosoy. We are committed to making breakthrough progress on AI safety, and with your support we believe we will have a significant and lasting impact. ## **References** * **\[Hoogland et al. 2024\]** The Developmental Landscape of In-Context Learning by **Jesse Hoogland (Timaeus)**, **George Wang (Timaeus)**, Matthew Farrugia-Roberts (then independent, now at Timaeus / Krueger Lab), Liam Carroll (then independent, now Timaeus / Gradient), Susan Wei (University of Melbourne), Daniel Murfet (University of Melbourne). \[[Arxiv](https://arxiv.org/abs/2402.02364); [Distillation](https://www.lesswrong.com/posts/Zza9MNA7YtHkzAtit/stagewise-development-in-neural-networks)\] * **\[Lau et al. 2023\]** Quantifying Degeneracy in Singular Models via the learning coefficient by Edmund Lau (University of Melbourne), **George Wang**, Daniel Murfet, Susan Wei. \[[Arxiv](https://arxiv.org/abs/2308.12108); [Distillation](https://www.lesswrong.com/posts/6g8cAftfQufLmFDYT/you-re-measuring-model-complexity-wrong)\] * **\[Furman & Lau 2024\]** Estimating the Local Learning Coefficient at Scale by **Zach Furman (Timaeus)** and Edmund Lau. \[[Arxiv](https://arxiv.org/abs/2402.03698)\] * **\[Chen et al. 2023\]** Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition by Zhongtian Chen (University of Melbourne), Edmund Lau, Jake Mendel (then independent, now at Apollo Research), Susan Wei, Daniel Murfet . \[[Arxiv](https://arxiv.org/abs/2310.06301); [Distillation](https://www.lesswrong.com/posts/jvGqQGDrYzZM4MyaN/growth-and-form-in-a-toy-model-of-superposition)\]11:T978,### Project summary During my MSc dissertation, after leveraging LLMs to identify domain owners by analysing scraped privacy policy texts, a new challenge emerged: the models struggled to accurately identify owners from non-English privacy policy texts by giving random names. This issue was evident across key evaluation metrics, including accuracy and, precision. My research addresses a critical gap in the field of AI Existential Safety: understanding and improving the safety of large language models (LLMs) in multilingual contexts. Current LLM evaluations are predominantly English based, leading to a narrow view of these models' safety and capabilities. This project seeks to expand the understanding of LLM safety across languages, exploring how token-based language encoding affects LLM reasoning, alignment, and robustness. The rapidly expanding capabilities of LLMs necessitate rigorous safety evaluations to prevent potential risks to global communities, especially those in non-English-speaking regions. Multilinguality adds complexity to AI safety challenges by creating multiple “versions” of safety that vary between languages. This research is motivated by a desire to build globally safe AI systems that honor diverse cultural norms while resisting adversarial manipulation. ### What are this project's goals? How will you achieve them? Research Objectives 1. Assess Multilingual Safety: Investigate how well existing LLM safety evaluations perform across languages, hypothesizing that substantial safety and bias variations will emerge. 2. Develop Cross-Language Safety Interventions: Design methodologies to reinforce safety and reduce vulnerabilities across different linguistic contexts. 3. Enhance Representation Engineering: Use insights from multilingual variations to advance the creation of a language-agnostic safety framework that could contribute to robust, jailbreak-resistant LLMs. ### How will this funding be used? It will be used to fund my study as DPhil student at University of Oxford. ### Who is on your team? What's your track record on similar projects? I will work alone under supervision. ### What are the most likely causes and outcomes if this project fails? These is no fail measurement as it will show how different languages in LLM might affect AI Alignment. ### How much money have you raised in the last 12 months, and from where? Nothing till now12:T56d,### _A note from JueYan Zhang, who recommended this grant:_ H was recently accepted into a Computer Science PhD program at a leading UK university, where they will explore institutional and technical models for equitable access to safe AI systems. This is a high-leverage opportunity, but comes with significant upfront costs: 1. Tuition fees of \~£33,370 for the first year 2. Living costs of \~£24,420 3. Student Visa fees of \~£6,500 £64,290 _1.3482 (GBP/USD)_ 1.01 (1% FX fee) = $87,543. **H is not eligible for UK student loans and faces a tight deadline to confirm enrollment.** They are applying for other sources of funding but cannot count on them materializing in time to start this fall. I was introduced to H through trusted mutual contacts. They’ve worked with UK DSIT, where their research has informed efforts on AI risk assessment. They’ve also held research and fellowship positions at the Oxford Martin School, GovAI, CSER, and IAPS. Their publications and policy papers have been relevant in discussions such as the UK AI Safety Summit. The financial constraints they are facing are real blockers to pursuing this opportunity. Given the relevance of their research to global public goods and the potential for their work to inform policy in key areas of AI governance, I am happy to recommend funding to support H’s education and continuing research.13:T2bf6,**Tldr:** CeSIA successfully advised YouTubers, generating millions of views on AI safety (notably with [one video](https://www.youtube.com/watch?v=ZP7T6WAK3Ow) which made 4M+ views). We seek funding for 1 FTE for 3-6 months to scale this content creator outreach and advisory model. ## **1\. Problem** The general public and journalists increasingly turn to online sources for information about AI, yet face a scarcity of accessible, high-quality content explaining AI risks. The extremely rapid advances in AI technologies make it particularly difficult for the public to form opinions about risks based on reliable resources. Despite pockets of excellent resources (like Rob Miles, Kurzgesagt, Lethal intelligence, Rational Animation), significant gaps persist in broad public awareness. These quality resources are generally limited to isolated and already convinced audiences. This knowledge deficit is particularly concerning because informed public understanding of large-scale AI risks is essential for public support of informed policy decisions—an area that remains severely under-resourced. ## **2\. Proposed solution** The political bottleneck for AI safety is not the production of knowledge about risks posed by AI to society—which is already plentiful—but rather the dissemination of this knowledge. It is extremely time-consuming and difficult for organizations producing knowledge on AI safety to build their own audience and achieve large-scale reach. Therefore, it is strategic for them to leverage existing platforms by connecting with quality content creators who already have millions of followers and enjoy trust from their audience. We propose mobilizing the talent of influential content creators (YouTubers, podcasters, etc.) to produce and disseminate engaging, informative content on AI safety. The goal is to educate a broad audience through various formats and channels they already consult, by encouraging content creators to cover AI risks and AI safety, and by providing them with scientific consulting to ensure accuracy and depth. Trust is a key factor: audiences often exhibit high trust in creators they follow, potentially making them more receptive to messages delivered through these channels compared to direct institutional communication or advertising. ## **3\. Track record & approach** Our initial efforts in France demonstrate the merits of this model. We collaborated/advised/gave interviews, which notably led to: * **Major success:** Provided scientific consulting (approx. 20 hours involving fact-checking, argument refinement, suggesting angles) for a [video](https://www.youtube.com/watch?v=ZP7T6WAK3Ow) by French YouTuber **Ego**, which garnered **over 4 million views** in its first months. * Additionally, we advised **IA Mania** (approx. 10 hours), resulting in [a video](https://www.youtube.com/watch?v=xPzKXDukO90) with 25k views. (This was our first try to advise youtubers, and at this point we were not aiming too high) ## **4\. Two approaches** We see 2 complementary strategies: 1. **Advisory & collaboration - our priority** * Identify and contact influential creators (initially English and French-speaking, potentially expanding to other European languages). * Provide tailored scientific consulting, connecting creators with relevant experts from our network to improve script accuracy, depth, and argumentation. We could also facilitate connections with figures like Yoshua Bengio or Stuart Russell where appropriate. * Encourage diverse formats, prioritizing science communicators and creators with large, engaged audiences. * This model positions us strategically, avoiding the high costs of direct content production or sponsorship while potentially achieving greater reach than traditional academic outreach. * We are also going to try to do the 80/20 of reaching out to documentary agencies or other types of influencers on other social media platforms. * Our objective is to preserve the general tone and approach of the creators we contact, encouraging them to address risks in their own original way, while informing their content with the scientific knowledge we provide. * **Estimated cost:** Funding for 1 FTE for 3-6 months ($18k - $36k) 2. **Direct sponsorships (probably not for this grant)** * Directly fund production by major creators if appropriate, and if resources allow (similar to [Open Philanthropy sponsoring](https://www.openphilanthropy.org/grants/kurzgesagt-video-creation-and-translation/) [Kurzgesagt](https://www.openphilanthropy.org/grants/?q=kurzgesagt)). * **Estimated cost:** Variable and potentially high (€50k-€1M+), requiring dedicated funding. ## **5\. Expected impact & Theory of change** **Scaling potential:** Our initial 100 hours yielded significant reach. With a dedicated 6-month FTE (\~900 hours), we could scale this effort significantly. At a greater scale, we anticipate greater efficiency through refined processes, established contacts, better targeting, and leveraging the increasing public interest in AI. **Forecast (based on past conversion rates & creator audience sizes):** For a 6-month FTE effort, we project the following potential reach within 12 months of content release: * 50% chance of generating >2 Million cumulative views (the median) * 25% chance of generating >4 Million cumulative views * 10% chance of generating >10 Million cumulative views * 1% chance of generating >50 Million cumulative views These estimates are conservative and take into account the potential "luck factor" of Ego's considerable video success. **Cost-effectiveness:** As a datapoint: Open Philanthropy has been donating roughly $2M per year to Kurzgesagt, for an output of around 10 videos per year on EA related topics (very broadly defined), each of them making around 5M views on average. That amounts to $0.04 per view (assuming this budget indeed supported these specific videos, but we don’t have the exact details of this sponsorship). Based on our projections, we expect our project to reach 2 million views over the next 12 months (median estimate), with an expected total around 3-4 million views when accounting for the possibility of much higher outcomes that pull the mean above the median. With a budget of $36k, this amounts to $0.01 per view. This is four times lower than our estimate for the Kurzgesagt sponsorships. Although individual video production might differ from Kurzgesagt's high-budget style, this approach strategically prioritizes reaching a wider, more diverse audience, mitigating the audience concentration typically seen with single-channel sponsorships. **Second-order benefits:** * **Spillover effects:** Quality arguments entering the public debate can influence traditional media. Following the Ego video, we were contacted by TF1 (France's main TV channel) for an internal presentation and by a TV documentary production agency. * **Report & A/B testing:** We plan systematic experimentation and reporting. We will test approaches with different creator types (e.g., science vs. commentary), messaging frames (e.g., technical vs. societal impact), and content formats (e.g., deep dives vs. short explainers) to identify what works best. We will track the number of collaborations established, analyse comments under the videos, track the media/influencer mentions generated (spillover), and finally will Report our findings from A/B testing. ## **6\. Risks and mitigations** **Fear mongering & paralysis:** While raising concerns is clearly the objective, communication research suggests that fear appeals require careful handling, often needing strong efficacy messages (solutions or actions people can take) to avoid defensive reactions or paralysis. CeSIA's consulting will need to guide creators not just on the _what_ (the risks) but also the _how_ (the framing and tone) to maximize constructive impact. We have experience doing that, and we course corrected the video of Ego for example to avoid a message that would have been to doomy/paralysing. **Ego being an extreme feat of luck:** While the Ego collaboration demonstrates feasibility, relying on landing such high-impact partnerships sporadically carries risk. However, successful YouTubers typically achieve highly predictable view counts in terms of orders of magnitude (for example, between 100k and 500k views for popular science communicators). This predictability provides a solid foundation for our outreach strategy. Sustaining a pipeline of collaborations will depend heavily on the FTE's networking skills. If one channel does not work, we will iterate until we are certain that the general approach is doomed. At least we will know. But for sure, we cannot guarantee the same degree of success; it's a bet. **Message quality, accuracy, and nuance:** This reliance on third-party creators means that we relinquishe significant control over the final message and its framing. Communicating complex, nuanced, and sometimes contested topics like AI alignment, capabilities forecasting, and existential risk through third-party creators carries inherent risks of simplification or inaccuracy. Creators, even with guidance, may struggle to convey technical details correctly or may frame issues in ways that deviate from our intended message. We believe we managed this successfully with Ego. It was more challenging with IA Mania because before our collaboration, his videos had a more accelerationist tone, but we are quite satisfied with the final video we produced with him. So, while these risks exist, we have managed them well so far. We expect approximately 1 video out of 6 to fall below CeSIA's standards for discourse quality, but we need to accept this risk; if it becomes too problematic, we can be more selective in our future collaborations. **Ethical considerations:** Providing significant expert consulting constitutes a "material connection" that necessitates disclosure under advertising standards prevalent in many regions (like the FTC guidelines in the US). Fortunately, we are offering our services pro-bono, and we aim to be transparent about our role in the creation of these videos. YouTubers have consistently treated us as expert friends and mentioned us in their video descriptions. We believe the risk in this area is minimal. ## **7\. Initial request** We request funding for **approach #1 (advisory & collaboration)**, as a pilot: * **Goal:** Cover the cost of **1 FTE for 3 to 6 months**. * **Activities:** Initiate contacts (English/French), test and refine the scientific consulting model, evaluate scaling potential. * **Candidate profile:** Requires strong communication skills, understanding of AI safety concepts (or ability to learn quickly), project management capabilities, and ideally experience with online media/creators. This person will be managed and mentored by experienced CeSIA staff. * **Budget:** * **Minimum (3 months FTE): $18,000** (less ideal due to ramp-up time) * **Target (6 months FTE): $36,000** (allows for significant testing and impact) This initial phase will provide crucial data to determine the feasibility and optimal strategy for a larger-scale, long-term effort.14:T13fb,### **Overview** AI companies are currently locked in a race to create a superhuman AI, and no one knows how to make the first superhuman AI not kill everyone. There’s little governmental oversight or public awareness that everyone’s lives are being risked. We think that without government intervention directed at solving the problem and ensuring no existentially dangerous AI is developed anywhere in the world, humanity is not likely to survive. We further think it is possible to [carefully](https://www.lesswrong.com/posts/jiYLFomPPePy85eN8/ai-pause-governance-advocacy-might-be-net-negative) but honestly inform the governments and the public of the problem and increase the chance of it being addressed. There are non-costly interventions that governments can make even before they fully agree with our understanding of the risk, that seem robustly helpful, e.g., because these interventions increase the chance of governments understanding the problem well and being able to address it directly later. Our nonprofit aims to improve institutional response to existential risk from AI by developing and testing messaging about it and launching campaigns to educate the general public and key stakeholders about AI and related risks. We think communicating the problem in simple and understandable, but valid (including in technicalities) language can help substantially. **What’s your plan?** With minor omissions, we plan to: * Do message testing (understand the variables, determine what successfully communicates core intuitions about the technical problem to various demographics or causes concern for valid reasons). * Core intuitions we’d want to communicate range from “Normal computer programs are human-made instructions for machines to follow, but modern AI systems are instead trillions of numbers; the algorithms these numbers represent are found by computers themselves and not designed, controlled, or understood by humans” to “AI developers know how to use a lot of compute to make AI systems generally better at achieving goals, but don’t know how to influence the goals AI systems are trying to pursue, especially as AI systems become human-level or smarter” to “If monkeys really want something, but humans really want something different, and humans don’t really care about monkeys, humans would usually get what they wanted even if it means monkeys don’t; if an AI system that’s better at achieving goals than humans doesn’t care about humans at all, it would get what it wants even if it means humans won’t get what they want. We should avoid developing superhuman systems that are misaligned with human values”. * Very different messages will work most efficiently to successfully produce technical understanding in very different people. * It might be good to simultaneously promote the government incentivizing (and not creating obstacles for) narrow and clearly beneficial uses of AI (such as in drug development), as opposed to general AI or research that shortens the timelines of general AI: we want regulation to target exclusively the inherently risky technology that might kill everyone. **Responsible, economically valuable innovation that doesn’t contribute to that risk, including lots of kinds of startups, should be supported**. * Iterate through content to promote with short feedback loops. * Experiment with various novel forms of messaging that have the potential to go viral. * Share results with other organizations in the space. * Scale up, improve understanding of AI and increase support for helpful policies among people whose understanding and support could be more important, coordinate with others in the space on various actions. * Prepare potential responses to possible future crises, and help people improve their understanding of AI and related risks by clearly and honestly communicating around major issues as they become public. **How will this funding be used?** Expenses by category depending on our overall annual budget: ![](https://fkousziwzbnkdkldjper.supabase.co/storage/v1/object/public/image-uploads/d9a1ea89-88c2-6d17-4237-ff055cd5bbca) **What's your track record?** Early testing showed that making the general public read technical explanations of x-risk from AI can cost as little as $0.10 per click; we’ve also had positive experience testing communicating about x-risk, including changing the minds of 3 out of 4 people we talked to at an e/acc event. **What do you plan to do about the risks?** It’s important to carefully monitor for risks throughout the work, e.g., use polls and look at the response to the messaging from various demographics and people with different priors to actively decrease polarization (and prevent the messaging from increasing it) and at the potential for backlash and have good feedback loops from that information; avoid anything in the lines of astroturfing; etc. **What other funding are you or your project getting?** We’ve received a speculation grant from the SFF; our application is currently being evaluated in the SFF’s s-process.15:T1711,### Project summary We are a team of four scholars at MATS, led by Alex Turner. We invented a method that grants highly-configurable control over how neural networks learn to represent and process information, whether that be at the level of parameters, activations, or modules. We have promising early results on several toy models, including MNIST autoencoders, small language models (Tinystories 8M, Qwen 0.5B), and a toy gridworld RL agent. We are currently scaling to a real-world unlearning problem. ### What are this project's goals? How will you achieve them? Our goal is to develop the method and demonstrate its efficacy-- as well as its relevance to AI risk-- to the broader research community. To do this, are pursuing three related research directions: * Feature localization and interpretability in LLMs: train LLMs in such a way to localize concepts and computation, exploring novel opportunities for interpretability and control (e.g. steering scalars; compare with ). These should produce tangible, interpretable subcomponents of networks or features in networks. * Deep unlearning / factored capabilities for LLMs: e.g. apply our methods to LLMs on the WMDP unlearning benchmark () and achieve state-of-the-art resistance to fine-tuning attacks, demonstrating “deep” unlearning without prohibitive costs to retain set performance. * Scalable oversight via localized motivations: reduce goal misgeneralization occurring under sparse behavioral feedback by localizing an agent’s “motivations” to specific, known components that generalize to cases without oversight. We intend to demonstrate this in simple RL environments simulating “partial oversight” (where a policy updates based on rewards that we can’t observe, see, e.g. ), then move to more complex environments (e.g. as in ). We aim to publish our initial findings in ICLR 2025, submission deadline: Oct 1, 2024. ### How will this funding be used? To rent reliable, always-on, remote-development machines with A100 GPUs with 80GB of VRAM until the ICLR submission deadline of Oct 1, so that we can experiment as rapidly as possible. With the minimum funding, we will be able to rent a reliable 8x A100 GPU machine until Sept. 8\. With the target funding, we will be able to continue to rent our current 4x A6000 machine from Sept 8 to Sept 29, plus a 8x A100 GPU machine from now until Sept 29. Note: we are exploring discounted compute options that wouldn’t compromise our rate of development and experimentation (potentially offered by Hofvarpnir Studios or FluidStack). ### Who is on your team? What's your track record on similar projects? Jacob Goldman-Wetzler: Published research on mixed precision neural nets with Lu Group at Yale () at age 17/18; many open-source contributions to Zig Programming Language compiler (); I was one of the youngest Recurse Center (writer’s retreat for programmers in NYC) participants at 13. Evžen Wybitul: Graduated top of class in Bioinformatics at Charles University, Prague, now enrolled in a Data Science Master’s programme at ETH Zurich. Worked professionally as a data scientist, then collaborated with Deepmind’s David Lindner on benchmarking safety-relevant capabilities of VLMs (work submitted to NeurIPS). Joseph Miller: Paper published at COLM 2024 on circuit finding in mechanistic interpretability (). Worked as Research Engineer at FAR AI on their first adversarial Go paper (). As a solo developer, created Hypnogram in 2021, a text-to-image AI website that generated over 5.4 million images for 1.2 million users (). Worked as a software engineer in big data for 1+ years. BS in Math and Computer Science. Alex Cloud: PhD in Statistics; 4 years of ML research in competitive industry jobs (Riot Games AI, Amazon): led research projects on novel applications to real-world games (e.g. , ); started undergrad data science mentorship program (), participated in MLAB 2.0. Alex Turner: MATS mentor and Research Scientist at Deepmind with a strong track record of successful mentorship and influential contributions to the AI safety literature. ### What are the most likely causes and outcomes if this project fails? The project has already succeeded in producing interesting and novel results. However, it may fail to improve outcomes from transformative AI if (i) the kind of structure we seek to impose on models inhibits their ability to fit data, reducing capabilities beyond a point of usefulness, or if (ii) low-alignment-tax localization could work for real-world problems and models in principle, but requires semantically disentangled data labels that are impractical to procure or estimate. ### What other funding are you or your project getting? We have a small compute budget from MATS (just above $5350 for our team of four) that we are currently on track to spend entirely by Sept 8, even without increasing our rate of experimentation. We have applied to the Long Term Future Fund and FAR AI for 6-month salaries and compute budget to cover our work during the MATS extension phase, but do not expect to hear back from them for another month. Also, even if we are funded, it is likely that our requested compute budget would not be sufficient to cover both accelerated experimentation now and 6 months of regular research activity.16:Tb0a,### Project summary My paper on [using access controls to address the dual-use nature of AI models](https://arxiv.org/abs/2505.09341) got accepted to ICML's technical AI governance workshop, and I seek funding to help me go there. I'm the sole author, so if I don't go, the work won't be presented. This would be my first conference ever, and it comes at a perfect time as I'm moving from technical AI safety to technical AI governance. ### What are this project's goals? How will you achieve them? I have two main goals. The main thing I'm excited about is the networking and learning opportunity. I'm mid-transition from technical AI safety to technical AI governance — getting to talk to people at the workshop would help me quickly understand what problems people are working on, who's doing what, and find collaborators for future work. Plus there's all that tacit knowledge you can't get from just reading papers. Since this is my first conference, I expect the impact to be pretty significant for my career direction. As for the paper itself — it's about the dual-use nature of AI models (how the same model can be used for good or harmful purposes) and why current approaches don't address this. I propose verification-based access controls as one potential solution, together with a possible technical implementation. The work is still pretty conceptual and needs many holes to be filled in by future work, but I do think something like access controls will eventually be part of AI systems, and maybe this paper can help start that conversation. I am a Master's student, and worked on the project myself, so I can't get any support from my university / lab. ### How will this funding be used? I just found out about the acceptance 2 days ago, so these are rough estimates: * Student conference ticket: $550 * Flight from Prague and back: \~$750 * Accommodation for 9 days (AirBnB within \~30min of venue): \~$1,200 (I'm looking into other, at least partial, funding sources, hence I set the minimum to $500) ### What are the most likely causes and outcomes if this project fails? The main way the first goal could "fail" is if I go but don't manage to connect with people. However, I will have a very thorough plan and specific talking points to help me resolve main cruxes about my future research directions, so I don't expect this failure mode to be likely. As for the paper, it might turn out that the practical obstacles to implementing access controls are just too big, which would limit its impact. But I think even learning that would be useful. ### How much money have you raised in the last 12 months, and from where? Last summer, I was part of Team Shard, which got compute funding from Neel Nanda/Manifund and general support funding from LTFF for our MATS project on gradient routing.17:T1095,### **Project Summary** --- ### **What are this project's goals? How will you achieve them?** I’m building GENESIS — a new AGI architecture that’s lightweight, transparent, and runs on everyday hardware instead of billion-dollar GPU clusters. My goal is to create an AI system whose reasoning is fully visible and traceable, so we can understand how it thinks, align it safely, and make advanced AI accessible to individuals and small teams, not just tech giants. Instead of relying on giant black-box neural nets, GENESIS uses symbolic memory structures and deterministic reasoning paths. Every step of its thinking can be inspected like code, making it fundamentally safer and more controllable than current LLMs. I’ve already built working prototypes proving the concept. I’m hitting: * Training speeds over **400,000 tokens/sec** * Custom RAG pipelines handling **1.3 million tokens/sec** * Symbolic inference speeds exceeding **2 million tokens/sec** My next steps are: * Refining the architecture * Building user-facing demos * Publishing technical documentation * Running external audits to prove safety and reproducibility If we succeed, GENESIS won’t just compete with LLMs — it will redefine what’s possible in safe, democratized AI. --- ### **How will this funding be used?** I’m seeking **$250,000** to **$350,000** to: * Fund 12–18 months of full-time development * Acquire hardware for testing and scaling prototypes * Build an open-source prototype with transparent reasoning chains * Develop live demos to show GENESIS answering complex questions with traceable logic paths * Hire occasional contractors for UI, optimization, or documentation work * Run external audits for safety and reproducibility * Cover legal, business, and miscellaneous costs My vision is to release GENESIS as a public resource so independent developers and researchers can build safe, powerful AI systems without needing massive infrastructure. --- ### **Who is on your team? What's your track record on similar projects?** Right now, it’s mostly me. I’m a technical polymath with over 20 years of experience in software engineering, algorithms, systems architecture, and signal processing. I’ve shipped large-scale software systems, built custom algorithms in domains like audio processing and AI, and run independent R&D projects from scratch. Over the past year, I’ve funded GENESIS out of pocket — writing prototypes, testing symbolic architectures, and achieving speeds competitive with mainstream AI systems. I’m obsessed with building things that actually work, and with understanding how they work under the hood. If funding comes through, I plan to add a small team to accelerate development and testing. --- ### **What are the most likely causes and outcomes if this project fails?** This is high-risk R&D. The biggest risk is that GENESIS may prove too complex or too resource-intensive to scale up to a practical, general-purpose AGI engine. Even if it falls short of fully replacing modern LLMs, I believe the project will still produce valuable outputs: * New tools for symbolic reasoning * Faster and more efficient retrieval systems * Safer architectural ideas for building explainable AI Worst-case, we’d have learned a great deal about how to combine symbolic reasoning with modern AI techniques, and I’d publish those results so others can build on the work. --- ### **How much money have you raised in the last 12 months, and from where?** I haven’t raised external funding yet. Everything so far has been bootstrapped from my personal savings. I’m actively applying to several grants (Emergent Ventures, Nonlinear, others) and exploring potential funding partnerships. Responses are in progress but no funds have landed yet. --- **Thanks for considering supporting GENESIS. I’m passionate about building AGI that’s safe, transparent, and truly for everyone. Happy to answer any technical questions or share demos.** Project Doc: Demo/Running code: 18:T4f46,## Project summary [Apart](https://apartresearch.com/) is an AI safety research organization that incubates talented researchers and facilitates innovative research in AI safety for hundreds of people around the globe through short [research sprints](https://alignmentjam.com/) and our Apart Lab [research fellowship](https://apartresearch.com/lab/fellowship). * [**Research Sprints (Alignment Jams)**](https://apartresearch.com/sprints/overview)**:** The Apart sprints offer weekend-long opportunities for talented individuals worldwide to engage in AI safety research and create original work. * [**Apart Lab Fellowship**](https://apartresearch.com/lab/fellowship)**:** The Apart Lab is a 3-6 month long online fellowship guiding promising teams of aspiring researchers to produce significant academic contributions showcased at world-leading AI safety conferences. Apart provides research mentorship and project management to participating teams. In 2023 alone, Apart has hosted 17 research sprints with more than 1000 participants and over 170 resulting projects; 10 of our 24 Apart Lab fellows have already completed their research, resulting in three publications accepted at top-tier academic ML workshops and conferences ([NeurIPS](https://arxiv.org/abs/2310.01870), [ACL](https://aclanthology.org/2023.findings-acl.733/), [ICLR](https://arxiv.org/abs/2305.19911)), and five currently under review ([1](https://arxiv.org/abs/2311.04131), [2](https://arxiv.org/abs/2310.08164), [3](https://arxiv.org/abs/2310.13121), two with restricted access); two of our fellows are on track to secure research positions with experienced AI safety researchers at Oxford University. **Apart Research Sprints** are * **Career-changing:** Sprints are focused on increasing participants' ability and confidence to contribute to AI safety research with a self-reported update towards working on AI safety of 9.15 percentage points (see [testimonials](https://apartresearch.com/sprints/overview#testimonials) for their why). * **Research incubators:** Sprints in 2022 and 2023 resulted in [more than 200 research projects](https://apartresearch.com/sprints/overview). See, for example, our sprint on [new failure cases in multi-agent AI systems](https://alignmentjam.com/jam/multiagent), resulting in 21 submitted projects, 2 workshop papers at NeurIPS ([1](https://openreview.net/forum?id=5HuBX8LvuT), [2](https://openreview.net/forum?id=HPmhaOTseN)), and 2 contributions to a major upcoming review on multi-agent security by CAIF. * **Global:** We offer our sprints both fully remotely and at more than [50 in-person locations](https://alignmentjam.com/map) across the globe – including top universities like Stanford, UC Berkeley, Cambridge, Oxford as well as comparatively underserved locations (India, Vietnam, Brazil, Kenya) – to invite a global and diverse research community to engage with AI safety. * **Community-driven:** With our decentralized approach, we make it easy for local organizers to run complex events (see [testimonials](https://alignmentjam.com/why)) and get together teams of researchers to think about the most important questions in AI safety (more than 200 research teams participated to date). **Apart Lab Fellowships** are * **Incubators for AI safety researchers:** Apart Lab fellows take charge of their own research projects and become first authors on their paper while being supported with research guidance and project management by experienced mentors (see [testimonials from fellows](https://apartresearch.com/lab#testimonials)). * **Providers of counterfactual career benefit:** We help fellows develop their research skills, connect with senior AI safety researchers (see [co-authors](https://apartresearch.com/research)), develop their research portfolio (in 2023, 3 conference papers + 5 papers under review) and secured impactful research roles (in 2023, two Apart Lab fellows on track to secure research positions with experienced AI safety researchers at Oxford University). * **Output-focused:** Apart Lab fellows conceive impactful AI safety research, and Apart Lab helps them make it real! We provide the structure of a lab environment, including internal peer review, deadlines for accountability, as well as a default for sharing information (subject to our [info-hazard policy](https://alignmentjam.com/publishing-policy)) with the research community (see [all research outputs](https://apartresearch.com/research)) — while prioritizing projects with impact. * **Remote-first:** Apart Lab fellowships are open to aspiring researchers globally without requiring relocation to London or Berkeley. **At Apart**, our work is * **Impact-focused:** Apart is dedicated to reducing existential risks from AI via AI safety technical research and governance, with a focus on empirical projects. Apart directly confronts AI safety challenges by facilitating research in mechanistic interpretability, safety evaluations, and conceptual alignment ([read more](https://apartresearch.com/research)) and by training the next generation of AI safety researchers. * **Filling important gaps:** We aim to scale up AI safety mentorship and make it accessible to everyone across the globe. We maintain a positive, solution-focused, open-minded culture to invite a diverse community to contribute to AI safety research. * **Embedded in the wider AI safety ecosystem:** We work with a wide array of research organizations such as [Apollo Research](https://www.apolloresearch.ai/), the [Cooperative AI Foundation](https://www.cooperativeai.com/foundation), the [Turing Institute](https://www.turing.ac.uk/), and [Oxford University](https://www.ox.ac.uk/) and have collaborated with researchers at [DeepMind](https://deepmind.google/) and [OpenAI](https://openai.com/). We also actively engage with the effective altruism community, e.g. by speaking at [EAGx](https://www.effectivealtruism.org/ea-global) conferences. ## What are this project's goals and how will you achieve them? Apart aims to reduce existential risk from AI by **facilitating new AI safety research projects** and **incubating the next generation of global AI safety researchers**. We achieve this by running global research sprints and an online research fellowship, which are neglected and impactful ways of incubating AI safety research projects and researchers from the global talent pool. This funding will allow us to improve the quality and scale of our incubation efforts over the next 6 months. **Expand the reach of our research sprints** * In the first half of 2024, we aim to double the number of participants in our AI safety research sprints by **engaging 1,000 new sprint participants**, achieving this milestone in half the time it took to reach our first 1,000 while maintaining our priority on participant quality. * Building on the success of our previous research sprints — with our Interpretability, Evaluations, Governance and Benchmarks events drawing from 60 to over 150 participants each — we will use the grant to **improve their reach and quality** by bringing in more partners (like [CAIF](http://cooperativeai.org/) and [Apollo Research](https://www.apolloresearch.ai/)), schedule our Sprints months in advance, advertising our sprints to more target groups, partnering with new local sites for the events, improving the infrastructure for the events and improving event mentorship and talks. * Given the current **abundance of work to be done in AI safety**, we want to actively explore new agendas in technical AI safety and AI Governance, where possible in cooperation with other organizations (as we have done before with[ Entrepreneur First](https://www.joinef.com/) and the[ Cooperative AI Foundation](https://www.cooperativeai.com/foundation)). **Bring more talented researchers into AI safety** * Having established [the Apart Lab](https://apartresearch.com/lab), we will increase capacity for the quantity and quality of mentorship to **support fellows in becoming active contributors** to impactful AI safety projects. Our focus extends to research agendas in technical governance and evaluations, in collaboration with partners specializing in applied alignment and AI security. Our goal is to produce outputs that are directly beneficial for governance and alignment, in addition to our academic publications. * We have clear signs – a good publication track record and early signs of being able to place Apart Lab graduates in impactful roles – that the Apart Lab fellowship can accelerate aspiring researchers' impact in AI safety research and help them pick up the skills they need to succeed. We will **invite 30 fellows** during the Spring 2024 cohorts and significantly **improve the quality of mentorship** by adding capacity and improving processes. * Our goal is to help aspiring AI safety researchers to get to a position where they can meaningfully contribute. We help transition skilled Apart Lab fellows into full-time roles when possible. This grant will also allow us to support the fellows' continued journey. ## How will this funding be used? We seek funding to **sustain and grow Apart as a whole**. This includes the costs to run our research sprints and research fellowships, support for Apart Lab research fellows, as well as costs for compute, contractors, conference attendance, and payroll. To **sustain** these current efforts for the **coming 6 months,** we calculate the following funding needs: * **Salary costs:** $105k * **Operational & Admin costs,** incl. software, outreach, fiscal sponsorship, workspace and other miscellaneous costs: $33k * **Research costs,** compute, APIs, conference travel: $26k To **grow** Apart further over the **coming 6 months,** we calculate the following funding needs: a) **Expand the Apart core team** (additional 2 FTE for six months; onboarding in process): $44k * 1FTE Ops and Fundraising support: $22k * 1FTE Research assistant for Apart Lab: $22k b) **Offer stipends to Apart Lab fellows** based on pre-determined milestones during the fellowship to allow our global community to dedicate more time to solving important issues within AI safety: $30k for 30 fellows during the coming six months at $1k / fellow. c) **Attract and remunerate external senior mentors** to provide support for our Apart Lab researchers, further improving the quality of our academic output: $10k for 200 mentor hours at \~$50/hour Based on the amount of funding received, we will prioritize allocating resources initially to maintain our operations, followed by investing in the growth of our organization, in accordance with the outlined priorities. ## Who is on your team and what's your track record on similar projects? Apart Research is led by [Esben Kran](https://www.notion.so/464e5bd35eea48e1b95adf9dc3ccd2ec?pvs=21) (Director) and [Jason Hoelscher-Obermaier](https://www.linkedin.com/in/jas-ho/?originalSubdomain=at) (Co-director). Our team members have previously worked for [ARC Evals](https://evals.alignment.org/) (now METR), [Aarhus University](https://chc.au.dk/), the [University of Vienna](https://www.univie.ac.at/en/), and 3 AI startups. Find an overview of the [Apart leadership on our website](https://apartresearch.com/about). Our success with both our Research Sprints and the Apart Lab fellowship is best showcased by the following milestones and achievements: **Research Sprint Achievements:** * Since November 2022, Apart hosted 19 research sprints with 1,248 participants and 209 submitted projects, with research from the sprints accepted at top-tier conferences and workshops * We have co-hosted and collaborated with many global research organizations, including [Apollo](https://www.apolloresearch.ai/), [CAIF](https://www.cooperativeai.com/foundation), and [Entrepreneur First](https://www.joinef.com/). Additionally, our Research Sprints hosted speakers from [DeepMind](https://deepmind.google/), [OpenAI](https://deepmind.google/), [Cambridge](https://www.cam.ac.uk/) and [NYU](https://www.nyu.edu/), among others * We provided real value in our collaboration and contracting for [CAIF](https://www.cooperativeai.com/foundation) on a Multi-Agent Risks research report, resulting from our research sprint with over **150 signups** and **21 submissions.** Their Research Director **Lewis Hammond** had the following feedback: > _We (the Cooperative AI Foundation) partnered with Apart Research to run a hackathon on multi-agent safety, to feed into an important report. We needed to work to tight deadlines but overall the process of organising the event was smooth, and there were many more participants than I was expecting. Of those I spoke to afterwards, everyone remarked on how much they enjoyed taking part._ * Our Research Sprints have a meaningful impact on participants with an **average net promoter score of 8.6/10** (indicating the likelihood of recommending the event to others) and 80% of participants reporting a **2x-10x or greater value in terms of counterfactual time spent**. Below are some testimonials from participants following our Interpretability Research Sprint: > _A great experience! A fun and welcoming event with some really useful resources for initiating interpretability research. And a lot of interesting projects to explore at the end!_ Alex Foote, MSc, Data Scientist - this project was incubated by Apart and eventually turned into a publication at ICLR 2023, with Alex as the Lead author > _The Interpretability Hackathon exceeded my expectations, it was incredibly well organized with an intelligently curated list of very helpful resources. I had a lot of fun participating and genuinely feel I was able to learn significantly more than I would have, had I spent my time elsewhere. I highly recommend these events to anyone who is interested in this sort of work!_ _Chris Mathwin, MATS scholar_ **Apart Lab Achievements:** * In 2023, Apart Lab fellows published 6 papers in total; among them 3 at [ACL](https://aclanthology.org/2023.findings-acl.733/) and workshop tracks of [NeurIPS](https://arxiv.org/abs/2310.01870) and [ICLR](https://arxiv.org/abs/2305.19911). Our publications have tackled model evaluations (e.g. the[ robustness of model editing techniques](https://aclanthology.org/2023.findings-acl.733/)) and mechanistic interpretability (e.g. of [interpreting intrinsic reward models learned by RLHF](https://arxiv.org/abs/2310.08164)), as well as providing accessible and scalable tools for probing model internals (e.g. [neuron2graph](https://arxiv.org/abs/2310.08164) and [DeepDecipher](https://arxiv.org/abs/2310.01870)). With more funding, we would invest more time into mentoring and establishing external collaborations to further improve the impact potential of our research outputs. * Two Apart Lab graduates are on track to secure research positions with experienced AI safety researchers at Oxford University. This demonstrates that Apart Lab can identify and mentor top talent. In the coming six months, we want to significantly increase the number of placements by improving our processes for follow-up support to our graduates and investing more time in follow-up mentoring. * Apart Lab also provided significant mentoring value to our fellows as evidenced in the following testimonials: > _As an undergraduate with little experience in academia, Apart was very helpful in guiding me through the process of improving my work through the publication process. They helped me to bridge the gap between a rough hackathon submission and a well-refined conference paper. I’d recommend the Apart Lab Fellowship for anyone looking to break into the research community and work on the pressing problems in AI Safety today."_ > > _"Like lots (most?) people at Apart Lab, I'm trying to transition from one career to another in AI Safety. There is no well-trodden "best" path so an organisation like Apart Lab which is willing to help individuals for free is a god send. My initial entry was the Hackathon - which was a tough weekend, as I came in with little AI knowledge, but I made some (little) progress, and received positive feedback from the moderators. \[...\] If (when!) I land a job in AI Safety it will be because of Apart Lab's help. It's a great organisation._ We aim to build on and improve upon our existing track record to incubate more high-impact AI safety researchers around the globe. ## What are the most likely causes and outcomes if this project fails? (premortem) We list the most likely possible causes of failures specific to our projects, together with the likely default outcomes and the mitigation strategies we will implement. * Info-hazards * **Outcome:** Publishing work that disproportionately helps dangerous capabilities compared to safety * **Mitigation strategies:** Define and refine our [publication policy](https://alignmentjam.com/publishing-policy) to have internal and external review for info hazards * Lack of research taste for Apart * **Outcome:** The events and projects from Apart focus on less impactful or even harmful research topics * **Mitigation strategies:** Obtain external feedback on Apart’s research portfolio; continue to collaborate with external advisors and researchers with complementary backgrounds; and continue exposing our research to scientific peer review * Lack of research taste for participants and fellows * **Outcome:** Aspiring researchers spend too much time on fruitless research * **Mitigation strategies:** Our mentors and collaborators continually discuss and evaluate research projects on informative review criteria for research taste and impact; mentor meetings focus on empowering and guiding the fellows' own research * Lab graduates do not realize their potential after the fellowship * **Outcome:** Lab fellows either get a non-AI-safety industry job or do not realize their potential within AI safety * **Mitigation strategies:** Besides the output-focused fellowship structure which helps fellows build legible AI safety credentials, we also stay in contact with and follow up with fellows to provide feedback and opportunities * Unintentional support of capability-focused organizations * **Outcome:** Our research or partnerships indirectly assists capability-oriented organizations * **Mitigation strategies:** We discuss our partnerships with external advisors and consider the implications of our projects on capabilities advancement (in addition to following our project publication policy) * De-motivating AI safety talent * **Outcome:** Talented and initially motivated individuals end up not contributing to AI safety research * **Mitigation strategies:** Gather anonymous feedback on the frequency and quality of mentorship, focusing on mentor availability and skill relevance; monitor and adjust project deadlines to ensure they are not jeopardizing mental health and work-life balance; clear communication around the academic review process to put potential rejections into context; and ensuring that research outputs' impact are communicated. * Poor selection of Apart Lab fellows * **Outcome:** We select fellows who do not have the right skills to contribute to AI safety research or cannot do so for logistical reasons * **Mitigation strategies:** Improve reach and targeting of sprints; improve our evaluation processes for Apart Lab candidates, focusing on the research and collaboration skills demonstrated during the sprint and during an initial trial phase. We believe that, given its past track record, Apart has a high chance of continued success if we devote sufficient attention to these potential risks and implement appropriate mitigation strategies. ### What other funding are you or your project getting? Apart receives several project-specific sponsorships or contractor assignments ranging from $1,000 to $20,000 that are outside of the scope of this grant. Previous sponsors include Apollo Research, EntrepreneurFirst, and CAIF. To support Apart’s continued development in impactful research we are seeking funding from currently open AI safety foundations, including CLR and EA Funds. We anticipate a high counterfactual value of the funding during the next 6 months.