Privítajte Mistral-sk-7b!
After a long time, the Slovak AI community got a new, this time a really big language model for the Slovak language. Almost three years after the release of the first Slovak language model, SlovakBERT, the collective of authors formed by Peter Bednár from the Department of Cybernetics and Artificial Intelligence FEI TUKE, Marek Dobeš from the Center for Social and Psychological Sciences SAV and Radovan Garabík from the Ľudovít Štúr Institute of Linguistics SAV used the Mistral multilingual model as a basis -7B-v0.1 with seven billion parameters.
Data from the web corpus "Araneum Slovacum VII Maximum" were used for fine-tuning of all parameters. As the authors state, if necessary, the quality of the pre-trained model can be improved by further training on a specific problem. At the same time, they point out that the model does not contain any moderating mechanism.
The training of the Mistral-sk-7b model was carried out on the currently seventh most powerful supercomputer in the world, Leonardo, procured by the consortium of the same name, of which Slovakia is also a member. The authors obtained the necessary computing time thanks to a successful project within the national challenge for access to the Leonardo supercomputer, which was coordinated by the Computing Center of the Slovak Academy of Sciences in cooperation with the National Supercomputer Center.
Model Mistral-sk-7b is published under the license "Apache 2.0" on the platform Hugging Face, under the organization "Slovak NLP Community"