Pi labs logo
Ship magical AIs faster by building with Pi's Scorers & Optimizers
  1. 1

    Step 1: Pi builds your scoring system to capture your application requirements.

    Pi reasons about your application to create a tunable scoring system that reflects your application’s specific success metrics.

    Provide Pi with a qualitative description of your application and Pi will generate your first scorer. First see how the scorer performs against a range of responses and then move to the next step to start using it to optimize your model quality.
    Google Colab Icon
    Pi scorers are modeled as a tree of metrics that assess and combine various dimensions of response quality.
  1. 2

    Step 2: Pi uses your scorer to optimize your prompt, model, and chain.

    Pi surveys a range of optimization techniques that help you raise quality of your application.

    1. Optimize your prompt

      Use a scorer to manually adjust your prompt and compare response changes

    2. Optimize your inference

      Use a scorer to optimally route your requests from smaller to bigger models

    3. Train your own model

      Use a scorer to filter and select data and track training progress across iterations.

    thumbnail
  2. 3

    Step 3: Great work. You’re now ready to unlock the full Pi toolkit.

    Pi gives you access to over 30 vetted ML and data science techniques, compiled against your scorer, packed behind easy to use Playgrounds and APIs.

    All difficulties
    Easy
    Medium
    Hard
    Playground
    Code
    Score responses
    Create a Scoring system
    Scoring calibration by raters
    Scoring calibration by user feedback
    Optimize your prompt
    Prompt iteration with scoring
    Automated iteration with DSPy
    Cross-model prompt migration
    Dynamic few shot optimization
    Customize your Search
    Query set generation
    Query Classification
    Query Fanout
    Custom Ranking
    Optimize your chain
    Ensemble generation with pruning
    Small to big model cascading
    Constrained generation
    Model routing
    Build your own models
    Supervised learning with Labels
    Supervised learning with Preferences
    Model distillation
    Reinforcement learning with GRPO
    Clean up & scale your data
    Data filtering
    Data clustering & labeling
    Synthetic data generation
    Web-based data acquisition
    Easy
    Create a Scoring system
    Transform subjective quality criteria into quantifiable metrics using data science-driven decomposition. Manually calibrate the importance of the different scoring dimensions.
    Jump in
    Code
    thumbnail
    Learn how it works