Sakana AI Announces AI CUDA Engineer That Can Speed Up Model Development and Deployment

Sakana AI, a Tokyo-based synthetic intelligence (AI) agency, launched a brand new synthetic intelligence (AI) agentic framework that may enhance the event and deployment speeds of enormous language fashions (LLMs). Announced on Thursday, the corporate unveiled the AI CUDA Engineer that improves each the pre-training and inference speeds of an AI mannequin by optimising the codebase. The AI agency highlighted that your entire course of is pushed by AI brokers and is end-to-end automated. Notably, Sakana AI launched The AI Scientist final yr which might conduct scientific analysis.

Sakana AI Unveils AI CUDA Engineer

In a put up, the Japanese AI agency said that after creating AI programs that may create new fashions, and totally automate the AI analysis course of, it started engaged on methods to hurry up the deployment and inference speeds of an LLM.

The firm mentioned that the analysis led to the event of the AI CUDA Engineer. It is a totally automated, complete agent framework for CUDA (Compute Unified Device Architecture) kernel discovery and optimisation.

CUDA kernels might be understood as specialised features that run on Nvidia GPUs, permitting parallel execution of code throughout a number of threads. Due to parallelism, it’s extra optimised than conventional strategies and permits for the acceleration of computational duties, particularly these with giant datasets. As such, that is thought of an effective way to optimise AI fashions’ deployment and inference.

Sakana AI mentioned the AI CUDA Engineer can routinely convert PyTorch modules into optimised CUDA kernels, to considerably enhance deployment speedups. It can generate kernels which might be mentioned to be 10-100 occasions sooner than its PyTorch counterpart.

The course of consists of 4 steps. First, the agent framework converts the PyTorch code into working kernels. Then, the agent implements optimisation strategies to make sure solely the most effective kernels are generated. Then, kernel crossover prompts are added, which mix a number of optimised kernels to create new kernels. Finally, the AI agent preserves the high-performance CUDA kernels in an archive, that are used to ship efficiency enhancements. The firm has additionally revealed a research that additional particulars the method.

Alongside the paper, Sakana AI can be publishing the AI CUDA Engineer Archive, which is a dataset consisting of greater than 30,000 kernels generated by the AI. These kernels are launched below the CC-By-4.0 license and might be accessed by way of Hugging Face.

Additionally, the Japanese agency additionally launched an internet site that lets guests interactively discover 17,000 verified kernels and their profiles. The web site permits customers to discover these kernels throughout 230 duties, and additionally lets them examine CUDA kernels throughout particular person experiments.