{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "87920721-d15a-4ad2-84c0-7b9e82df176f",
   "metadata": {},
   "source": [
    "# Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b88b066b-3d5f-4603-a49f-e94b06b97860",
   "metadata": {},
   "source": [
    "## Importing packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec38e9d6-c56e-4703-976f-3264b7c37fb6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "377a9b2f-527a-4610-acb7-bb1261da6642",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tmplot as tmp\n",
    "import pickle as pkl\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aebd7122-44e0-46f9-b999-400c6930be48",
   "metadata": {},
   "source": [
    "## Importing data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd9119c3-e46d-4870-b066-d0866810e45a",
   "metadata": {},
   "source": [
    "Let's take the BTM model trained on a test dataset (*SearchSnippets*) as an example.\n",
    "We will begin with reading it from a file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "292c08e3-0fa3-431c-bd6e-b414f086dd28",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('data/model_btm.pkl', 'rb') as file:\n",
    "    model = pkl.load(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63773c31-8b45-4e20-9b55-a61cb5f7987c",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = pd.read_csv('data/SearchSnippets.txt.gz', header=None).values.ravel()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7483f62b-ffc6-466a-bb5e-71375c29bd4d",
   "metadata": {},
   "source": [
    "## Matrices"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47eb8fbf-7327-42ef-9e27-fe523518dd0f",
   "metadata": {},
   "source": [
    "Researchers working with topic models often need to obtain `phi` (words vs topics probability) and `theta` (topics vs documents probability) matrices.\n",
    "*Tmplot* provides two functions for getting these matrices from `tomotopy`, `bitermplus`, and `gensim` models."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80d6fd7a-b600-4331-b1bb-0bf68932faf9",
   "metadata": {},
   "source": [
    "### Phi matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77650819-89bb-4764-86b6-ce615a86e83d",
   "metadata": {},
   "source": [
    "Note that you will need to pass a vocabulary for a gensim model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c13528d2-0663-42e6-b08a-1461dfd4eaf3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>topics</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>words</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>aaa</th>\n",
       "      <td>3.195102e-08</td>\n",
       "      <td>3.012856e-08</td>\n",
       "      <td>3.047842e-08</td>\n",
       "      <td>3.542745e-08</td>\n",
       "      <td>3.836165e-08</td>\n",
       "      <td>2.961217e-08</td>\n",
       "      <td>2.362519e-08</td>\n",
       "      <td>4.831267e-08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aaas</th>\n",
       "      <td>3.837318e-05</td>\n",
       "      <td>3.012856e-08</td>\n",
       "      <td>3.047842e-08</td>\n",
       "      <td>3.542745e-08</td>\n",
       "      <td>3.836165e-08</td>\n",
       "      <td>5.922729e-04</td>\n",
       "      <td>6.144912e-05</td>\n",
       "      <td>2.903592e-05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aaron</th>\n",
       "      <td>3.195102e-08</td>\n",
       "      <td>3.012856e-08</td>\n",
       "      <td>3.047842e-08</td>\n",
       "      <td>3.542745e-08</td>\n",
       "      <td>4.296888e-04</td>\n",
       "      <td>2.961217e-08</td>\n",
       "      <td>2.362519e-08</td>\n",
       "      <td>4.831267e-08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aau</th>\n",
       "      <td>3.195102e-08</td>\n",
       "      <td>3.012856e-08</td>\n",
       "      <td>3.047842e-08</td>\n",
       "      <td>3.542745e-08</td>\n",
       "      <td>3.836165e-08</td>\n",
       "      <td>2.961217e-08</td>\n",
       "      <td>2.362519e-08</td>\n",
       "      <td>4.203686e-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>abbreviations</th>\n",
       "      <td>7.990951e-05</td>\n",
       "      <td>3.163800e-04</td>\n",
       "      <td>3.047842e-08</td>\n",
       "      <td>3.542745e-08</td>\n",
       "      <td>3.836165e-08</td>\n",
       "      <td>2.961217e-08</td>\n",
       "      <td>2.386144e-06</td>\n",
       "      <td>4.831267e-08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "topics                    0             1             2             3  \\\n",
       "words                                                                   \n",
       "aaa            3.195102e-08  3.012856e-08  3.047842e-08  3.542745e-08   \n",
       "aaas           3.837318e-05  3.012856e-08  3.047842e-08  3.542745e-08   \n",
       "aaron          3.195102e-08  3.012856e-08  3.047842e-08  3.542745e-08   \n",
       "aau            3.195102e-08  3.012856e-08  3.047842e-08  3.542745e-08   \n",
       "abbreviations  7.990951e-05  3.163800e-04  3.047842e-08  3.542745e-08   \n",
       "\n",
       "topics                    4             5             6             7  \n",
       "words                                                                  \n",
       "aaa            3.836165e-08  2.961217e-08  2.362519e-08  4.831267e-08  \n",
       "aaas           3.836165e-08  5.922729e-04  6.144912e-05  2.903592e-05  \n",
       "aaron          4.296888e-04  2.961217e-08  2.362519e-08  4.831267e-08  \n",
       "aau            3.836165e-08  2.961217e-08  2.362519e-08  4.203686e-04  \n",
       "abbreviations  3.836165e-08  2.961217e-08  2.386144e-06  4.831267e-08  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "phi = tmp.get_phi(model)\n",
    "phi.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4c6f33a-0a87-4fcb-87ee-ee11a16d2ee8",
   "metadata": {},
   "source": [
    "### Theta matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81c8eeed-2bd9-43cb-9288-963348d45a46",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>docs</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>990</th>\n",
       "      <th>991</th>\n",
       "      <th>992</th>\n",
       "      <th>993</th>\n",
       "      <th>994</th>\n",
       "      <th>995</th>\n",
       "      <th>996</th>\n",
       "      <th>997</th>\n",
       "      <th>998</th>\n",
       "      <th>999</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>topics</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.354702</td>\n",
       "      <td>0.294777</td>\n",
       "      <td>0.178074</td>\n",
       "      <td>0.332888</td>\n",
       "      <td>0.596412</td>\n",
       "      <td>0.726975</td>\n",
       "      <td>0.099094</td>\n",
       "      <td>0.257602</td>\n",
       "      <td>0.532725</td>\n",
       "      <td>0.471059</td>\n",
       "      <td>...</td>\n",
       "      <td>0.007651</td>\n",
       "      <td>0.085897</td>\n",
       "      <td>0.025840</td>\n",
       "      <td>0.019194</td>\n",
       "      <td>0.033898</td>\n",
       "      <td>0.020408</td>\n",
       "      <td>0.030728</td>\n",
       "      <td>0.036133</td>\n",
       "      <td>0.084323</td>\n",
       "      <td>0.024301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.000245</td>\n",
       "      <td>0.007173</td>\n",
       "      <td>0.021324</td>\n",
       "      <td>0.019411</td>\n",
       "      <td>0.029472</td>\n",
       "      <td>0.008740</td>\n",
       "      <td>0.011804</td>\n",
       "      <td>0.036323</td>\n",
       "      <td>0.011349</td>\n",
       "      <td>0.003909</td>\n",
       "      <td>...</td>\n",
       "      <td>0.069988</td>\n",
       "      <td>0.263869</td>\n",
       "      <td>0.058431</td>\n",
       "      <td>0.227196</td>\n",
       "      <td>0.022920</td>\n",
       "      <td>0.021660</td>\n",
       "      <td>0.040932</td>\n",
       "      <td>0.060534</td>\n",
       "      <td>0.150018</td>\n",
       "      <td>0.071271</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.003073</td>\n",
       "      <td>0.057144</td>\n",
       "      <td>0.013837</td>\n",
       "      <td>0.014514</td>\n",
       "      <td>0.011813</td>\n",
       "      <td>0.002588</td>\n",
       "      <td>0.000247</td>\n",
       "      <td>0.027391</td>\n",
       "      <td>0.002325</td>\n",
       "      <td>0.005435</td>\n",
       "      <td>...</td>\n",
       "      <td>0.007558</td>\n",
       "      <td>0.014669</td>\n",
       "      <td>0.014206</td>\n",
       "      <td>0.002697</td>\n",
       "      <td>0.008854</td>\n",
       "      <td>0.017299</td>\n",
       "      <td>0.014710</td>\n",
       "      <td>0.027672</td>\n",
       "      <td>0.061375</td>\n",
       "      <td>0.011318</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.003678</td>\n",
       "      <td>0.029281</td>\n",
       "      <td>0.010010</td>\n",
       "      <td>0.001287</td>\n",
       "      <td>0.027349</td>\n",
       "      <td>0.004351</td>\n",
       "      <td>0.018189</td>\n",
       "      <td>0.085879</td>\n",
       "      <td>0.011453</td>\n",
       "      <td>0.002965</td>\n",
       "      <td>...</td>\n",
       "      <td>0.007010</td>\n",
       "      <td>0.022462</td>\n",
       "      <td>0.007516</td>\n",
       "      <td>0.006018</td>\n",
       "      <td>0.001193</td>\n",
       "      <td>0.007400</td>\n",
       "      <td>0.007335</td>\n",
       "      <td>0.021119</td>\n",
       "      <td>0.012309</td>\n",
       "      <td>0.006168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.000927</td>\n",
       "      <td>0.035162</td>\n",
       "      <td>0.001736</td>\n",
       "      <td>0.319421</td>\n",
       "      <td>0.024606</td>\n",
       "      <td>0.042996</td>\n",
       "      <td>0.019524</td>\n",
       "      <td>0.036119</td>\n",
       "      <td>0.001910</td>\n",
       "      <td>0.039332</td>\n",
       "      <td>...</td>\n",
       "      <td>0.016587</td>\n",
       "      <td>0.056386</td>\n",
       "      <td>0.005925</td>\n",
       "      <td>0.003503</td>\n",
       "      <td>0.001620</td>\n",
       "      <td>0.006468</td>\n",
       "      <td>0.004151</td>\n",
       "      <td>0.018374</td>\n",
       "      <td>0.008712</td>\n",
       "      <td>0.087364</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 1000 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "docs         0         1         2         3         4         5         6    \\\n",
       "topics                                                                         \n",
       "0       0.354702  0.294777  0.178074  0.332888  0.596412  0.726975  0.099094   \n",
       "1       0.000245  0.007173  0.021324  0.019411  0.029472  0.008740  0.011804   \n",
       "2       0.003073  0.057144  0.013837  0.014514  0.011813  0.002588  0.000247   \n",
       "3       0.003678  0.029281  0.010010  0.001287  0.027349  0.004351  0.018189   \n",
       "4       0.000927  0.035162  0.001736  0.319421  0.024606  0.042996  0.019524   \n",
       "\n",
       "docs         7         8         9    ...       990       991       992  \\\n",
       "topics                                ...                                 \n",
       "0       0.257602  0.532725  0.471059  ...  0.007651  0.085897  0.025840   \n",
       "1       0.036323  0.011349  0.003909  ...  0.069988  0.263869  0.058431   \n",
       "2       0.027391  0.002325  0.005435  ...  0.007558  0.014669  0.014206   \n",
       "3       0.085879  0.011453  0.002965  ...  0.007010  0.022462  0.007516   \n",
       "4       0.036119  0.001910  0.039332  ...  0.016587  0.056386  0.005925   \n",
       "\n",
       "docs         993       994       995       996       997       998       999  \n",
       "topics                                                                        \n",
       "0       0.019194  0.033898  0.020408  0.030728  0.036133  0.084323  0.024301  \n",
       "1       0.227196  0.022920  0.021660  0.040932  0.060534  0.150018  0.071271  \n",
       "2       0.002697  0.008854  0.017299  0.014710  0.027672  0.061375  0.011318  \n",
       "3       0.006018  0.001193  0.007400  0.007335  0.021119  0.012309  0.006168  \n",
       "4       0.003503  0.001620  0.006468  0.004151  0.018374  0.008712  0.087364  \n",
       "\n",
       "[5 rows x 1000 columns]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp.get_theta(model).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e406b148-2b5a-43d1-92ae-9b372e668759",
   "metadata": {},
   "source": [
    "## Documents"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8726b24b-127d-422c-9966-910a19edd62e",
   "metadata": {},
   "source": [
    "Here is how you can get documents with maximum probabilities $P(t|d)$ for each topic:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2f7dbde-d01b-40f6-8cd6-229c2941400c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>topic0</th>\n",
       "      <th>topic1</th>\n",
       "      <th>topic2</th>\n",
       "      <th>topic3</th>\n",
       "      <th>topic4</th>\n",
       "      <th>topic5</th>\n",
       "      <th>topic6</th>\n",
       "      <th>topic7</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>speakeasy speedtest speakeasy speed test test ...</td>\n",
       "      <td>links jstor sici sici jstor postwar consumptio...</td>\n",
       "      <td>imdb name julia roberts julia roberts imdb mov...</td>\n",
       "      <td>guitars bodies amps guitars strings</td>\n",
       "      <td>vcic unc edu vcic venture capital investment c...</td>\n",
       "      <td>washington edu drivers device drivers device d...</td>\n",
       "      <td>apache api dom document document xml standard ...</td>\n",
       "      <td>hypotheses hypotheses author illustrates hypot...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>speedtest bandwidth speed test bandwidth speed...</td>\n",
       "      <td>econpapers repec article econpapers postwar co...</td>\n",
       "      <td>celebrities cruise celebrity tom cruise tom cr...</td>\n",
       "      <td>louis french fashion designer designer manufac...</td>\n",
       "      <td>national venture capital association foster un...</td>\n",
       "      <td>manufactures parallel serial drives</td>\n",
       "      <td>schools dom default xml dom tutorial xml docum...</td>\n",
       "      <td>surreal surreal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>home bandwidth broadband speedtest bandwidth c...</td>\n",
       "      <td>findarticles articles consumption consumer exp...</td>\n",
       "      <td>imdb name tom cruise tom cruise imdb movies ce...</td>\n",
       "      <td>fashion designers default fashion designers fa...</td>\n",
       "      <td>san jose mercury news venture capital expanded...</td>\n",
       "      <td>leonardo leonardo vinci inventor information c...</td>\n",
       "      <td>access cards ieee access</td>\n",
       "      <td>allposters surrealism posters surrealism poste...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>home bandwidth broadband speedtest bandwidth c...</td>\n",
       "      <td>financial financial international health insur...</td>\n",
       "      <td>absolutely roberts absolutely julia roberts ph...</td>\n",
       "      <td>fashion designers audio fashion designer net f...</td>\n",
       "      <td>seattlepi nwsource venture seattle venture cap...</td>\n",
       "      <td>journals searching biomedical journals engine ...</td>\n",
       "      <td>generator xml generator sample xml instance do...</td>\n",
       "      <td>hypotheses hypotheses nature research hypothes...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>portfolio shareholder services manage investme...</td>\n",
       "      <td>consumption consumer rights consumption consum...</td>\n",
       "      <td>imdb title imdb movies celebs</td>\n",
       "      <td>fashion fashion designers fashion designers fa...</td>\n",
       "      <td>venture capital journal listening model ventur...</td>\n",
       "      <td>lwn articles driver lwn device drivers kernel ...</td>\n",
       "      <td>reference standard template library standard t...</td>\n",
       "      <td>allposters beatles posters beatles prints allp...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              topic0  \\\n",
       "0  speakeasy speedtest speakeasy speed test test ...   \n",
       "1  speedtest bandwidth speed test bandwidth speed...   \n",
       "2  home bandwidth broadband speedtest bandwidth c...   \n",
       "3  home bandwidth broadband speedtest bandwidth c...   \n",
       "4  portfolio shareholder services manage investme...   \n",
       "\n",
       "                                              topic1  \\\n",
       "0  links jstor sici sici jstor postwar consumptio...   \n",
       "1  econpapers repec article econpapers postwar co...   \n",
       "2  findarticles articles consumption consumer exp...   \n",
       "3  financial financial international health insur...   \n",
       "4  consumption consumer rights consumption consum...   \n",
       "\n",
       "                                              topic2  \\\n",
       "0  imdb name julia roberts julia roberts imdb mov...   \n",
       "1  celebrities cruise celebrity tom cruise tom cr...   \n",
       "2  imdb name tom cruise tom cruise imdb movies ce...   \n",
       "3  absolutely roberts absolutely julia roberts ph...   \n",
       "4                     imdb title imdb movies celebs    \n",
       "\n",
       "                                              topic3  \\\n",
       "0               guitars bodies amps guitars strings    \n",
       "1  louis french fashion designer designer manufac...   \n",
       "2  fashion designers default fashion designers fa...   \n",
       "3  fashion designers audio fashion designer net f...   \n",
       "4  fashion fashion designers fashion designers fa...   \n",
       "\n",
       "                                              topic4  \\\n",
       "0  vcic unc edu vcic venture capital investment c...   \n",
       "1  national venture capital association foster un...   \n",
       "2  san jose mercury news venture capital expanded...   \n",
       "3  seattlepi nwsource venture seattle venture cap...   \n",
       "4  venture capital journal listening model ventur...   \n",
       "\n",
       "                                              topic5  \\\n",
       "0  washington edu drivers device drivers device d...   \n",
       "1               manufactures parallel serial drives    \n",
       "2  leonardo leonardo vinci inventor information c...   \n",
       "3  journals searching biomedical journals engine ...   \n",
       "4  lwn articles driver lwn device drivers kernel ...   \n",
       "\n",
       "                                              topic6  \\\n",
       "0  apache api dom document document xml standard ...   \n",
       "1  schools dom default xml dom tutorial xml docum...   \n",
       "2                          access cards ieee access    \n",
       "3  generator xml generator sample xml instance do...   \n",
       "4  reference standard template library standard t...   \n",
       "\n",
       "                                              topic7  \n",
       "0  hypotheses hypotheses author illustrates hypot...  \n",
       "1                                   surreal surreal   \n",
       "2  allposters surrealism posters surrealism poste...  \n",
       "3  hypotheses hypotheses nature research hypothes...  \n",
       "4  allposters beatles posters beatles prints allp...  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp.get_top_docs(docs, model=model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcb89b61-e611-44cf-b264-08bee8df2746",
   "metadata": {},
   "source": [
    "## Visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de40a65f-d213-4e58-b92a-52fad8a75034",
   "metadata": {},
   "source": [
    "*tmplot* takes much from [LDAvis](https://github.com/cpsievert/LDAvis), but also extends the functionality with a number of algorithms and metrics for plotting topics and terms. *tmplot* is based on [ipywidgets](https://ipywidgets.readthedocs.io/) and [Altair](https://altair-viz.github.io/) ([Vega](https://vega.github.io/)-backed package for nice plots)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdf7fe50-601a-431f-bf33-f6de019eda7e",
   "metadata": {},
   "source": [
    "### Topics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f7b8d80-db66-4062-b878-3a8fee0e20ce",
   "metadata": {},
   "source": [
    "First, we need to calculate the coordinates of topics based on intertopic distance values. By default, the combination of *t-distributed Stochastic Neighbor Embedding* and *symmetric Kullback-Leibler divergence* is used to calculate topics coordinates in 2D, but a number of other metrics and algorithms are also available (see `tmplot.get_topics_dist` and `tmplot.get_topics_scatter` functions for additional information). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ea2a69d-cc8a-4545-a407-fd02bc69138a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>topic</th>\n",
       "      <th>size</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-41.183987</td>\n",
       "      <td>-30.480648</td>\n",
       "      <td>0</td>\n",
       "      <td>21.160233</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-11.704910</td>\n",
       "      <td>-34.631725</td>\n",
       "      <td>1</td>\n",
       "      <td>4.265470</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-56.292171</td>\n",
       "      <td>-4.832846</td>\n",
       "      <td>2</td>\n",
       "      <td>20.599346</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9.921317</td>\n",
       "      <td>-14.181945</td>\n",
       "      <td>3</td>\n",
       "      <td>7.176289</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-45.702721</td>\n",
       "      <td>22.987968</td>\n",
       "      <td>4</td>\n",
       "      <td>4.535249</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           x          y  topic       size  label\n",
       "0 -41.183987 -30.480648      0  21.160233      0\n",
       "1 -11.704910 -34.631725      1   4.265470      1\n",
       "2 -56.292171  -4.832846      2  20.599346      2\n",
       "3   9.921317 -14.181945      3   7.176289      3\n",
       "4 -45.702721  22.987968      4   4.535249      4"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topics_coords = tmp.prepare_coords(model)\n",
    "topics_coords.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3200724e-18f1-4f84-b751-400ee7c42323",
   "metadata": {},
   "source": [
    "Plotting topics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d40b041-7e75-48e0-89b2-7d60301cb7a9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<div id=\"altair-viz-79711ace9762469ab7821eb8f867758b\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-79711ace9762469ab7821eb8f867758b\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-79711ace9762469ab7821eb8f867758b\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm//vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm//vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm//vega-lite@4.8.1?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm//vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function loadScript(lib) {\n",
       "      return new Promise(function(resolve, reject) {\n",
       "        var s = document.createElement('script');\n",
       "        s.src = paths[lib];\n",
       "        s.async = true;\n",
       "        s.onload = () => resolve(paths[lib]);\n",
       "        s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "      });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else if (typeof vegaEmbed === \"function\") {\n",
       "      displayChart(vegaEmbed);\n",
       "    } else {\n",
       "      loadScript(\"vega\")\n",
       "        .then(() => loadScript(\"vega-lite\"))\n",
       "        .then(() => loadScript(\"vega-embed\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"config\": {\"view\": {\"continuousWidth\": 400, \"continuousHeight\": 300, \"stroke\": \"transparent\"}, \"axis\": {\"labelFontSize\": 13, \"titleFontSize\": 13}, \"legend\": {\"labelFontSize\": 13, \"orient\": \"bottom\", \"titleFontSize\": 13}}, \"layer\": [{\"mark\": \"rule\", \"encoding\": {\"color\": {\"value\": \"gray\"}, \"size\": {\"value\": 0.2}, \"y\": {\"type\": \"quantitative\", \"aggregate\": \"average\", \"field\": \"y\"}}}, {\"mark\": \"rule\", \"encoding\": {\"color\": {\"value\": \"gray\"}, \"size\": {\"value\": 0.2}, \"x\": {\"type\": \"quantitative\", \"aggregate\": \"average\", \"field\": \"x\"}}}, {\"mark\": {\"type\": \"circle\", \"opacity\": 0.33, \"stroke\": \"black\", \"strokeWidth\": 1}, \"encoding\": {\"size\": {\"type\": \"quantitative\", \"field\": \"size\", \"scale\": {\"range\": [0, 3000]}, \"title\": \"Marginal topic distribution\"}, \"tooltip\": [{\"type\": \"quantitative\", \"field\": \"label\"}, {\"type\": \"quantitative\", \"field\": \"size\"}], \"x\": {\"type\": \"quantitative\", \"axis\": null, \"field\": \"x\"}, \"y\": {\"type\": \"quantitative\", \"axis\": null, \"field\": \"y\"}}}, {\"mark\": {\"type\": \"text\", \"align\": \"center\", \"baseline\": \"middle\"}, \"encoding\": {\"size\": {\"value\": 13}, \"text\": {\"type\": \"quantitative\", \"field\": \"topic\"}, \"tooltip\": [{\"type\": \"quantitative\", \"field\": \"label\"}, {\"type\": \"quantitative\", \"field\": \"size\"}], \"x\": {\"type\": \"quantitative\", \"axis\": null, \"field\": \"x\"}, \"y\": {\"type\": \"quantitative\", \"axis\": null, \"field\": \"y\"}}}], \"data\": {\"name\": \"data-b4b8d52e6617f557c46e85af861098b2\"}, \"$schema\": \"https://vega.github.io/schema/vega-lite/v4.8.1.json\", \"datasets\": {\"data-b4b8d52e6617f557c46e85af861098b2\": [{\"x\": -41.18398666381836, \"y\": -30.480648040771484, \"topic\": 0, \"size\": 21.160233354931293, \"label\": 0}, {\"x\": -11.704910278320312, \"y\": -34.6317253112793, \"topic\": 1, \"size\": 4.265469601093625, \"label\": 1}, {\"x\": -56.292171478271484, \"y\": -4.832846164703369, \"topic\": 2, \"size\": 20.599346427153606, \"label\": 2}, {\"x\": 9.921317100524902, \"y\": -14.181944847106934, \"topic\": 3, \"size\": 7.176289186756818, \"label\": 3}, {\"x\": -45.702720642089844, \"y\": 22.98796844482422, \"topic\": 4, \"size\": 4.535248821274001, \"label\": 4}, {\"x\": -22.124332427978516, \"y\": -1.9493522644042969, \"topic\": 5, \"size\": 11.990284323265692, \"label\": 5}, {\"x\": 7.433163642883301, \"y\": 15.483391761779785, \"topic\": 6, \"size\": 21.70431027549243, \"label\": 6}, {\"x\": -17.32845115661621, \"y\": 32.00534439086914, \"topic\": 7, \"size\": 8.56881801003254, \"label\": 7}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.LayerChart(...)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp.plot_scatter_topics(topics_coords, size_col='size', label_col='label')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7567f299-40b3-446b-9080-3b768b7f894d",
   "metadata": {},
   "source": [
    "### Words (or terms)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23466ca3-6714-44fd-8fe1-20d2d771a02e",
   "metadata": {},
   "source": [
    "**tmplot** also uses terms *relevance* that was introduced by [Sievert and Shirley (2014)](https://www.aclweb.org/anthology/W14-3110.pdf) for sorting terms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91eb40cc-23ab-4c3f-808f-a4a109222bdc",
   "metadata": {},
   "outputs": [],
   "source": [
    "terms_probs = tmp.calc_terms_probs_ratio(phi, topic=0, lambda_=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cf53d3d-d3ad-4871-9af2-afb1b6d30c9b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<div id=\"altair-viz-599bcf1b492f4529bcfe78536e7a8c96\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-599bcf1b492f4529bcfe78536e7a8c96\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-599bcf1b492f4529bcfe78536e7a8c96\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm//vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm//vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm//vega-lite@4.8.1?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm//vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function loadScript(lib) {\n",
       "      return new Promise(function(resolve, reject) {\n",
       "        var s = document.createElement('script');\n",
       "        s.src = paths[lib];\n",
       "        s.async = true;\n",
       "        s.onload = () => resolve(paths[lib]);\n",
       "        s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "      });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else if (typeof vegaEmbed === \"function\") {\n",
       "      displayChart(vegaEmbed);\n",
       "    } else {\n",
       "      loadScript(\"vega\")\n",
       "        .then(() => loadScript(\"vega-lite\"))\n",
       "        .then(() => loadScript(\"vega-embed\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"config\": {\"view\": {\"continuousWidth\": 400, \"continuousHeight\": 300}, \"axis\": {\"labelFontSize\": 13, \"titleFontSize\": 13}, \"legend\": {\"columns\": 1, \"labelFontSize\": 13, \"labelLimit\": 250, \"titleFontSize\": 13}}, \"data\": {\"name\": \"data-183e3f1348034f183bc7333604d57afa\"}, \"mark\": \"bar\", \"encoding\": {\"color\": {\"type\": \"nominal\", \"field\": \"Type\", \"legend\": {\"orient\": \"bottom\"}, \"scale\": {\"scheme\": \"category20\"}}, \"x\": {\"type\": \"quantitative\", \"field\": \"Probability\", \"stack\": null}, \"y\": {\"type\": \"nominal\", \"field\": \"Terms\", \"sort\": null, \"title\": null}}, \"$schema\": \"https://vega.github.io/schema/vega-lite/v4.8.1.json\", \"datasets\": {\"data-183e3f1348034f183bc7333604d57afa\": [{\"Terms\": \"business\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.0359131477370519}, {\"Terms\": \"market\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.016653077871314915}, {\"Terms\": \"news\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.08217218242003155}, {\"Terms\": \"com\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.0968799499499213}, {\"Terms\": \"stock\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.01347665577913991}, {\"Terms\": \"services\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.015444668552024693}, {\"Terms\": \"finance\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.010758552186818105}, {\"Terms\": \"information\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.06580581710949925}, {\"Terms\": \"home\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.03712697648571675}, {\"Terms\": \"jobs\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.00987942839490457}, {\"Terms\": \"internet\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.018803530504891273}, {\"Terms\": \"engine\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.010815927854932312}, {\"Terms\": \"products\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.012993203885793716}, {\"Terms\": \"financial\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.009081312565508638}, {\"Terms\": \"marketing\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.007967964129337462}, {\"Terms\": \"yahoo\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.024280753032918576}, {\"Terms\": \"income\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.006404573783956772}, {\"Terms\": \"exchange\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.0073510584775152025}, {\"Terms\": \"tax\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.006810112565449422}, {\"Terms\": \"electrical\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.007958465001267827}, {\"Terms\": \"online\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.03940372542760388}, {\"Terms\": \"trade\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.015916538750742067}, {\"Terms\": \"security\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.008853860643887865}, {\"Terms\": \"search\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.01880882547178773}, {\"Terms\": \"motor\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.006178139509060493}, {\"Terms\": \"service\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.00963319386585024}, {\"Terms\": \"gov\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.033811048431616836}, {\"Terms\": \"management\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.012121299416280502}, {\"Terms\": \"data\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.015608465411704549}, {\"Terms\": \"company\", \"Type\": \"Marginal term probability, p(w)\", \"Probability\": 0.007582393828758683}, {\"Terms\": \"business\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.02009722504411948}, {\"Terms\": \"market\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.01469111205309877}, {\"Terms\": \"news\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.014352431215599837}, {\"Terms\": \"com\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.013122316852985786}, {\"Terms\": \"stock\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.012269224554757343}, {\"Terms\": \"services\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.009106073336606929}, {\"Terms\": \"finance\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.00904217129179581}, {\"Terms\": \"information\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.008645978613966869}, {\"Terms\": \"home\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.008342443901114051}, {\"Terms\": \"jobs\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.007805666724700649}, {\"Terms\": \"internet\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.007719398964205637}, {\"Terms\": \"engine\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.007706618555243414}, {\"Terms\": \"products\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.007157060969867786}, {\"Terms\": \"financial\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.007137890356424451}, {\"Terms\": \"marketing\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.006498869908313255}, {\"Terms\": \"yahoo\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.006393431534374907}, {\"Terms\": \"income\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.006374260920931572}, {\"Terms\": \"exchange\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.006345505000766568}, {\"Terms\": \"tax\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.006316749080601564}, {\"Terms\": \"electrical\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.006224091115625442}, {\"Terms\": \"online\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.0057831670064287174}, {\"Terms\": \"trade\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.005706484552655374}, {\"Terms\": \"security\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.005636192303363142}, {\"Terms\": \"search\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.005470046986854231}, {\"Terms\": \"motor\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.005396559635321444}, {\"Terms\": \"service\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.005300706568104765}, {\"Terms\": \"gov\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.005156926967279746}, {\"Terms\": \"management\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.0049716110373274995}, {\"Terms\": \"data\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.004946050219403052}, {\"Terms\": \"company\", \"Type\": \"Conditional term probability, p(w|t)\", \"Probability\": 0.004773514698413029}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.Chart(...)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp.plot_terms(terms_probs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79c7c6c0-46e1-40e3-bdef-c3b8eae78a99",
   "metadata": {},
   "source": [
    "### Documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e205a32c-efb1-4de7-a4b4-e647fa901033",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>topic0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>speakeasy speedtest speakeasy speed test test ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>speedtest bandwidth speed test bandwidth speed...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              topic0\n",
       "0  speakeasy speedtest speakeasy speed test test ...\n",
       "1  speedtest bandwidth speed test bandwidth speed..."
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_docs_topic0 = tmp.get_top_docs(docs, model=model, docs_num=2, topics=[0])\n",
    "top_docs_topic0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4786d0a0-f09e-4885-b374-e23cf53e83bb",
   "metadata": {},
   "source": [
    "The following output is used within the interactive interface that we will explore shortly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbe87e1f-e8e4-4cad-a51b-f19e1ee7c73a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>table td{text-align: left !important}table th{text-align: center !important}</style><table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>topic0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>speakeasy speedtest speakeasy speed test test speed internet connection speakeasy speed test</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>speedtest bandwidth speed test bandwidth speed test bandwidth bandwidth speed internet service</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp.plot_docs(top_docs_topic0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acc84831-c2c1-4c92-90b8-f87bd05e428e",
   "metadata": {},
   "source": [
    "## Interactive report interface"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90e235e4-ab52-465b-a394-7feee3a24827",
   "metadata": {},
   "source": [
    "To run the report interface, just call `tmplot.report()` function with your model and docs. You can tweak most of the hidden parameters using keyword arguments (see function docstring)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f443db82-1715-4d8e-9d0e-5a74a2e919c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp.report(model, docs=docs, height=400, width=250)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "751407f4-5f79-4331-b7a0-a141efd09d72",
   "metadata": {},
   "source": [
    "![Report](_static/topics_terms_plots.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}