Microsoft launched the household of Phi-3.5 synthetic intelligence (AI) fashions on Tuesday, because the successor of the Phi-3 fashions launched in April. The brand new launch contains Phi-3.5 Combination of Consultants (MoE), Phi-3.5 Imaginative and prescient, and Phi-3.5 Mini fashions. These are instruct fashions, so they won’t work as a typical conversational AI however would require customers so as to add particular directions to get the specified output. The open-source AI fashions can be found to obtain from the tech big’s Hugging Face listings.
Microsoft Releases Phi-3.5 AI Fashions
The discharge of the brand new AI fashions was introduced by Microsoft government Weizhu Chen in a publish on X (previously referred to as Twitter). The Phi-3.5 fashions provide upgraded capabilities over the predecessor, however the structure, dataset and coaching strategies largely stay the identical. The Mini mannequin has been up to date with multilingual assist, and the MoE and Imaginative and prescient fashions are new inclusions within the AI mannequin household.
Coming to technicalities, the Phi-3.5 Mini has 3.8 billion parameters. It makes use of the identical tokeniser (a instrument that breaks down textual content into smaller items) and a dense decoder-only transformer. The mannequin solely helps textual content as enter and helps a context window of 1,28,000 tokens. The corporate claims it was educated utilizing 3.4 trillion tokens between June and August, and its data cut-off is October 2023.
One key spotlight of this mannequin is that it now helps a number of new languages together with Arabic, Chinese language, Czech, Danish, Dutch, English, Finnish, French, German, Hebrew, Hungarian, Italian, Japanese, Korean, Norwegian, Polish, Portuguese, Russian, Spanish, Swedish, Thai, Turkish, and Ukrainian.
The Phi-3.5 Imaginative and prescient AI mannequin has 4.2 billion parameters and it consists of a picture encoder that enables it to course of info inside a picture. With the identical context size because the Mini mannequin, it accepts each textual content and pictures as enter. It was educated between July and August on 500 billion tokens of information and has a textual content data cutoff of March.
Lastly, the Phi-3.5 MoE AI mannequin has 16×3.8 billion parameters. Nonetheless, solely 6.6 billion of them are energetic parameters when utilizing two consultants. Notably, MoE is a method the place a number of fashions (consultants) are educated independently after which mixed to enhance the accuracy and effectivity of the mannequin. This mannequin was educated on 4.9 trillion tokens of information between April and August, and it has a data cutoff date of October 2023.
On efficiency, Microsoft shared benchmark scores of all the particular person fashions, and based mostly on the information shared, the Phi-3.5 MoE outperforms each Gemini 1.5 Flash and GPT-4o mini within the SQuALITY benchmark which checks the readability and accuracy when summarising an extended block of textual content. This checks the lengthy context window of the AI mannequin.
Nonetheless, it ought to be talked about that it’s not a good comparability since MoE fashions use a unique structure and require extra space for storing and extra refined {hardware} to run. Individually, the Phi-3.5 Mini and Imaginative and prescient fashions have additionally outperformed related competing AI fashions in the identical section in some metrics.
These serious about attempting out the Phi-3.5 AI fashions can entry them through Hugging Face listings. Microsoft mentioned that these fashions use flash consideration which would require customers to run the techniques on superior GPUs. The corporate has examined them on Nvidia A100, A6000, and H100 GPUs.
