Instructions to use microsoft/phi-2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use microsoft/phi-2 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="microsoft/phi-2")# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2") model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2") - Inference
- Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use microsoft/phi-2 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "microsoft/phi-2" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "microsoft/phi-2", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/microsoft/phi-2
- SGLang
How to use microsoft/phi-2 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "microsoft/phi-2" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "microsoft/phi-2", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "microsoft/phi-2" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "microsoft/phi-2", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use microsoft/phi-2 with Docker Model Runner:
docker model run hf.co/microsoft/phi-2
License
Hi,
Congrats on the release! In a developer presentation, Satya Nadella mentioned Phi 2 would be open sourced. Unfortunately, the Microsoft Research License doesn't seem to qualify as an open sourced license according to the OSI's Open Source Definition.
Might it be possible to switch the license to one approved by the OSI?
Thank you!
Section 1C of the license states "Models. If machine learning model(s) are included, you may use the model(s), but you may not distribute the models."
Does this inhibit anyone from finetuning the model and uploading it to hugging face? Need to know not sure if I'd be breaching user agreement, by furthering the development and research of this model.
Its a travesty to call this model 'open source', under current licensing. Only if Microsoft would respond the way mistral did, when asked about their licensing.
Hmm, I guess that means you can't finetune the model? So sad - there is so much potential in this model. The open source community could build on this model to create incredible things, like what OpenHermes did to Mistral.
I believe the constraint is in the training data used to build the model. MS has opened other models, but this one is using copywritten textbook data to guide the generation of the synthetic dataset [I think! not sure, waiting for an answer from Sebastien Bubeck
Wrong? So far as I know. Unless you count advancing research as commercial activity. They're not serving inference to customers in production with it, which is I think what is meant by commercial.
Related to #16
It switched to MIT!!!
