Developing a Transform Project

This guide assumes you have deployed your Transform project and downloaded your Transform git repo to your local machine. If you have not completed these steps, check out the deployment guide.

First, open a new terminal window and navigate to your Transform directory. We have a CLI interface that will allow you to query metrics from your Transform server. You can read more about the CLI interface here.

We recommend installing the Transform CLI in a virtual environment to avoid any python environment dependency issues.

Run the following commands to set up a virtual environment and activate it. If you don't want to check your virtual environment into git, you can add the directory to your .gitignore file.

python3 -m venv venv && source venv/bin/activate

Next, install Transform.

pip install transform

Finally, you need to authenticate to your Transform server. Run the following command and follow the prompts in your terminal. You will need your Transform API key for this step. You can view and manage your Transform API keys here.

mql setup

To view the available commands for the CLI run

mql --help

    /opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/site-packages/fuzzywuzzy/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
      warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
    Usage: mql [OPTIONS] COMMAND [ARGS]...
    
    Options:
      -v, --verbose
      -y, --skip-confirm
      --debug-log-file
      --help              Show this message and exit.
    
    Commands:
      commit-configs              Commit yaml configs found in specified...
      contact                     Instructions for how to contact Transform...
      data-warehouse-validations  Run data warehouse validations for a model,...
      drop-cache                  Drop the MQL cache.
      drop-materialization        ***NEW*** Create a new MQL drop...
      get-dimension-values        List all dimension values that are...
      health-report               Completes a health check on MQL servers.
      identify                    Identify the currently authenticated user.
      install-completion          Install command completion for the MQL CLI
      invalidate-caches           Invalidates caches
      latest-mql-image            Outputs the latest MQL server image details
      list-dimensions             List all unique dimensions for the...
      list-materializations       List the materializations for the...
      list-metrics                List the metrics for the Organization with...
      list-queries                Retrieve queries from mql server
      list-servers                Lists available MQL servers.
      materialize                 Create a new MQL materialization query,...
      pin-model                   Pin a model id from configs that are...
      ping                        Perform basic HTTP health check against...
      query                       Create a new MQL query, polls for...
      setup                       Guides user through CLI setup.
      stream-query-logs           Retrieve queries from mql server
      unpin-model                 Unpin a model id
      validate-configs            Validate yaml configs found in specified...
      version                     Print the current version of the MQL CLI.

You should now be ready to query metrics and develop Transform locally.

Getting Started on the Transform CLI Workflow with Transform

Organization of your Transform repo

As long as you are within the overarching repo folder that contains the relevant YAML files to build the Transform model, you can organize your YAML files however you would like (e.g. different subfolders to break out metrics and data sources). You cannot, however, have other non-Transform YAML files present in the repo (think of the Transform repo as THE model that looks "down" at all levels beneath it. For more context, see the Transform Model.)

CLI Workflow with Transform

Navigate to the repo that holds the YAML files of your Transform model. Make sure that you have Transform installed, or your virtual environment is activated.

Quick useful commands:

mql identify: This command helps you identify the current model you are working on + when the model was last updated.

mql list-metrics: This command lists the metrics in your Transform model

mql list-dimensions: This command lists the dimensions available to your metric(s)

• ex: I want to know all available dimensions in the metric named active_user.
  • The CLI command is mql list-dimensions --metric-names active_user
• ex: I want to know all available dimensions for metrics active_user and inactive_user.
  • The CLI command is mql list-dimensions --metric-names active_user,inactive_user
  • Note that list-dimensions gives you the common dimensions between your metrics of interest.

The general workflow to edit, test, and make changes to your Transform model:

1. Switch to your relevant github branch
2. Make your edits to the appropriate YAML files
3. mql commit-configs --config-dir .
   • This step validates the YAML files in your current directory (specified via --config-dir .), checking for syntax errors and incorrect parameters.
   • When you commit configs, you will be asked whether you want to pin your model (enter y for yes, n for no). Pinning a model means that you are now running a "local" Transform model with your changes. Once your model is pinned, you can now query against your local model. For a quick tutorial on querying, see the MetricFlow tutorial (Transform customers will use the mql command, and not mf).
   • You may also include --skip-dw to skip validating against your warehouse to check if columns exist (ex: mql commit-configs --skip-dw --config-dir .)
4. Once satisfied, you can push your changes to your git repo, and merge your branch.
   • For Transform customers: once your changes are merged into the GitHub Repo, the GitHub Actions will automatically trigger to update the model on the Transform app. Once the Github Actions finish, the updated model should also be reflected in the Transform App.
5. Remember to unpin your model via mql unpin-model (once unpinned, your local configuration will point to whatever is the repo's last-updated model).
   • Tip: mql identify will show you the current model you are on via model ID, as well as the timestamp as to when the model was updated.