{
 "cells": [
  {
   "id": "md-000",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Representation as Features\n",
    "## Dimensionality Reduction, Projection, Embeddings and Logistic Scores\n",
    "\n",
    "**Warin Wattanapornprom, PhD. · Data Science and Engineering Teaching Lab**\n",
    "\n",
    "Notebook นี้ออกแบบให้นักศึกษารันตามลำดับ ทดลองเปลี่ยน parameter และตอบคำถามจากผลที่เห็นจริง\n",
    "\n",
    "> จุดประสงค์ไม่ใช่หาเทคนิคที่ “ดีที่สุด” แต่เรียนรู้ว่า representation แต่ละแบบเก็บอะไร ทิ้งอะไร ใช้ข้อมูลแบบใด และจะป้องกัน leakage อย่างไร\n",
    "\n",
    "### Learning outcomes\n",
    "1. แยก unsupervised, supervised และ model-derived representations\n",
    "2. Fit transformer เฉพาะ training data\n",
    "3. เปรียบเทียบ PCA, SVD, NMF, ICA, random projection, hashing, LDA, PLS, autoencoder, UMAP และ t-SNE\n",
    "4. สร้าง logistic score และ out-of-fold stacking features\n",
    "5. ประเมิน reconstruction, neighborhood preservation และ downstream performance\n",
    "6. ออกแบบ output contract สำหรับนำ representation ไปใช้ซ้ำ\n"
   ]
  },
  {
   "id": "md-001",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0 · Environment and reproducibility\n",
    "\n",
    "Notebook หลักใช้ NumPy, pandas, matplotlib และ scikit-learn ส่วน UMAP/PyTorch เป็น optional cells หากเครื่องไม่มี package จะข้ามอย่างสุภาพ\n"
   ]
  },
  {
   "id": "code-002",
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment and run once when your environment does not have the packages.\n",
    "# In Jupyter/Colab, remove the leading # from the next line:\n",
    "# %pip install -q numpy pandas matplotlib scikit-learn umap-learn\n",
    "\n",
    "# PyTorch is optional and is usually preinstalled in Google Colab.\n",
    "# See https://pytorch.org/get-started/locally/ for the command matching your system.\n"
   ]
  },
  {
   "id": "code-003",
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys, time, warnings\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn import __version__ as sklearn_version\n",
    "from sklearn.datasets import load_breast_cancer, load_digits, make_regression\n",
    "from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_predict\n",
    "from sklearn.preprocessing import StandardScaler, MinMaxScaler\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import accuracy_score, log_loss, roc_auc_score\n",
    "\n",
    "SEED = 42\n",
    "rng = np.random.default_rng(SEED)\n",
    "print(\"Python\", sys.version.split()[0], \"| scikit-learn\", sklearn_version)\n"
   ]
  },
  {
   "id": "md-004",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 · Dataset and the non-negotiable split\n",
    "\n",
    "เราใช้ Breast Cancer dataset เพื่อเปรียบเทียบหลาย representation บนโจทย์ classification เดียวกัน\n",
    "\n",
    "**กฎ:** split ก่อน fit scaler, reducer หรือ supervised transformer ทุกครั้ง\n"
   ]
  },
  {
   "id": "code-005",
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = load_breast_cancer(as_frame=True)\n",
    "X, y = data.data, data.target\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=.25, stratify=y, random_state=SEED\n",
    ")\n",
    "print(X_train.shape, X_test.shape, y_train.value_counts(normalize=True).round(3).to_dict())\n"
   ]
  },
  {
   "id": "code-006",
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_representation(name, Z_train, Z_test):\n",
    "    model = LogisticRegression(max_iter=3000, random_state=SEED)\n",
    "    started = time.perf_counter()\n",
    "    model.fit(Z_train, y_train)\n",
    "    p = model.predict_proba(Z_test)[:, 1]\n",
    "    elapsed = time.perf_counter() - started\n",
    "    return {\"representation\": name, \"dimensions\": Z_train.shape[1],\n",
    "            \"auc\": roc_auc_score(y_test, p), \"log_loss\": log_loss(y_test, p),\n",
    "            \"model_seconds\": elapsed}\n",
    "\n",
    "results = []\n",
    "baseline = Pipeline([(\"impute\", SimpleImputer(strategy=\"median\")),\n",
    "                     (\"scale\", StandardScaler())])\n",
    "Xb_train = baseline.fit_transform(X_train)\n",
    "Xb_test = baseline.transform(X_test)\n",
    "results.append(evaluate_representation(\"Scaled raw\", Xb_train, Xb_test))\n",
    "pd.DataFrame(results)\n"
   ]
  },
  {
   "id": "md-007",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 · PCA — variance-preserving linear components\n",
    "\n",
    "PCA เหมาะกับ continuous correlated features แต่ component ไม่ได้มีชื่อทาง domain โดยอัตโนมัติ\n",
    "\n",
    "ลองเปลี่ยน `n_components` เป็น 2, 5, 10 และ `.95` แล้วดู AUC, explained variance และเวลา\n"
   ]
  },
  {
   "id": "code-008",
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "\n",
    "pca_pipe = Pipeline([(\"impute\", SimpleImputer(strategy=\"median\")),\n",
    "                     (\"scale\", StandardScaler()),\n",
    "                     (\"pca\", PCA(n_components=.95, svd_solver=\"full\"))])\n",
    "Zp_train = pca_pipe.fit_transform(X_train)\n",
    "Zp_test = pca_pipe.transform(X_test)\n",
    "results.append(evaluate_representation(\"PCA 95% variance\", Zp_train, Zp_test))\n",
    "print(\"shape:\", Zp_train.shape,\n",
    "      \"variance:\", pca_pipe[-1].explained_variance_ratio_.sum().round(4))\n"
   ]
  },
  {
   "id": "code-009",
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca = pca_pipe[-1]\n",
    "plt.figure(figsize=(8, 3))\n",
    "plt.plot(np.cumsum(pca.explained_variance_ratio_), marker=\"o\")\n",
    "plt.xlabel(\"Number of retained components\")\n",
    "plt.ylabel(\"Cumulative explained variance\")\n",
    "plt.grid(alpha=.25); plt.show()\n"
   ]
  },
  {
   "id": "md-010",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Try it**\n",
    "- PCA 2 มิติให้ภาพที่ดีแต่ Classification ดีพอหรือไม่?\n",
    "- Feature ใดมี loading สูงใน PC1 และควรตั้งชื่อ PC1 หรือไม่?\n",
    "- ถ้าไม่ Standardize ผลจะเปลี่ยนอย่างไร?\n"
   ]
  },
  {
   "id": "md-011",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 · Truncated SVD / LSA — sparse representation\n",
    "\n",
    "สร้าง corpus ขนาดเล็กใน Notebook จึงไม่พึ่ง Internet แล้วลด TF–IDF เป็น latent dimensions\n"
   ]
  },
  {
   "id": "code-012",
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.decomposition import TruncatedSVD\n",
    "\n",
    "docs = [\n",
    " \"data pipeline warehouse orchestration\", \"etl airflow batch pipeline\",\n",
    " \"neural network image classification\", \"deep learning representation\",\n",
    " \"stock return volatility liquidity\", \"financial risk market price\",\n",
    " \"protein sequence kmer bioinformatics\", \"gene expression rna structure\"\n",
    "]\n",
    "labels = [\"data\",\"data\",\"ai\",\"ai\",\"finance\",\"finance\",\"bio\",\"bio\"]\n",
    "tfidf = TfidfVectorizer(ngram_range=(1,2))\n",
    "Xs = tfidf.fit_transform(docs)\n",
    "svd = TruncatedSVD(n_components=4, random_state=SEED)\n",
    "Zs = svd.fit_transform(Xs)\n",
    "print(\"TF-IDF\", Xs.shape, \"-> LSA\", Zs.shape)\n",
    "pd.DataFrame(Zs.round(3), index=labels, columns=[f\"LSA{i+1}\" for i in range(4)])\n"
   ]
  },
  {
   "id": "md-013",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4 · NMF — non-negative parts-based factors\n",
    "\n",
    "NMF เหมาะกับ count/intensity/non-negative matrices และมักอ่าน top loading ได้ง่ายกว่า signed components\n"
   ]
  },
  {
   "id": "code-014",
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import NMF\n",
    "\n",
    "nmf = NMF(n_components=4, init=\"nndsvda\", max_iter=1000, random_state=SEED)\n",
    "W = nmf.fit_transform(Xs)\n",
    "H = nmf.components_\n",
    "terms = np.array(tfidf.get_feature_names_out())\n",
    "for k, row in enumerate(H):\n",
    "    print(f\"factor {k+1}:\", \", \".join(terms[row.argsort()[-5:][::-1]]))\n",
    "print(\"reconstruction error:\", round(nmf.reconstruction_err_, 4))\n"
   ]
  },
  {
   "id": "md-015",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5 · ICA — recover statistically independent sources\n",
    "\n",
    "Synthetic experiment ผสม sine wave สองแหล่ง แล้วให้ FastICA พยายามแยกกลับ\n"
   ]
  },
  {
   "id": "code-016",
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import FastICA\n",
    "\n",
    "t = np.linspace(0, 8, 2000)\n",
    "S_true = np.c_[np.sin(2*t), np.sign(np.sin(3*t))]\n",
    "S_true += .05 * rng.normal(size=S_true.shape)\n",
    "A = np.array([[1.0, .5], [.4, 1.2]])\n",
    "X_mix = S_true @ A.T\n",
    "ica = FastICA(n_components=2, whiten=\"unit-variance\", max_iter=1000,\n",
    "              random_state=SEED)\n",
    "S_hat = ica.fit_transform(X_mix)\n",
    "\n",
    "fig, ax = plt.subplots(2, 1, figsize=(10, 5), sharex=True)\n",
    "ax[0].plot(t[:300], X_mix[:300]); ax[0].set_title(\"Observed mixtures\")\n",
    "ax[1].plot(t[:300], S_hat[:300]); ax[1].set_title(\"Recovered independent components\")\n",
    "plt.tight_layout(); plt.show()\n"
   ]
  },
  {
   "id": "md-017",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6 · Random Projection — fast distance-preserving compression\n",
    "\n",
    "ตรวจ Pairwise Distance distortion จาก sample ก่อนตัดสินใช้กับ downstream model\n"
   ]
  },
  {
   "id": "code-018",
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.random_projection import GaussianRandomProjection\n",
    "from sklearn.metrics import pairwise_distances\n",
    "\n",
    "rp = GaussianRandomProjection(n_components=12, random_state=SEED)\n",
    "Zr_train = rp.fit_transform(Xb_train)\n",
    "Zr_test = rp.transform(Xb_test)\n",
    "results.append(evaluate_representation(\"Random projection\", Zr_train, Zr_test))\n",
    "\n",
    "idx = rng.choice(len(Xb_train), 80, replace=False)\n",
    "d0 = pairwise_distances(Xb_train[idx])\n",
    "d1 = pairwise_distances(Zr_train[idx])\n",
    "mask = d0 > 0\n",
    "print(\"median relative distance error:\", np.median(np.abs(d1[mask]-d0[mask])/d0[mask]))\n"
   ]
  },
  {
   "id": "md-019",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7 · Feature Hashing — bounded-memory sparse features\n",
    "\n",
    "เหมาะกับ open vocabulary และ streaming แต่ collision ต้องเป็นส่วนหนึ่งของการออกแบบ\n"
   ]
  },
  {
   "id": "code-020",
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction import FeatureHasher\n",
    "\n",
    "events = [\n",
    " {\"country=TH\":1, \"device=mobile\":1, \"page=/course\":1},\n",
    " {\"country=JP\":1, \"device=desktop\":1, \"page=/research\":1},\n",
    " {\"country=TH\":1, \"device=mobile\":1, \"page=/research\":1}\n",
    "]\n",
    "hasher = FeatureHasher(n_features=16, input_type=\"dict\", alternate_sign=True)\n",
    "Xhash = hasher.transform(events)\n",
    "print(Xhash.shape, \"non-zero:\", Xhash.nnz)\n",
    "print(Xhash.toarray())\n"
   ]
  },
  {
   "id": "md-021",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8 · LDA — supervised discriminant components\n",
    "\n",
    "LDA ใช้ label จึงต้อง fit ภายใน training fold เท่านั้น จำนวนแกนสูงสุดคือ classes − 1\n"
   ]
  },
  {
   "id": "code-022",
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis\n",
    "\n",
    "lda = LinearDiscriminantAnalysis(solver=\"eigen\", shrinkage=\"auto\", n_components=1)\n",
    "Zl_train = lda.fit_transform(Xb_train, y_train)\n",
    "Zl_test = lda.transform(Xb_test)\n",
    "results.append(evaluate_representation(\"LDA component\", Zl_train, Zl_test))\n",
    "print(Zl_train.shape, \"test accuracy:\", lda.score(Xb_test, y_test))\n"
   ]
  },
  {
   "id": "md-023",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9 · PLS — response-aware latent scores\n",
    "\n",
    "ใช้ Regression dataset เพื่อให้เห็นว่า PLS เลือก Component ที่สัมพันธ์กับ Y ไม่ใช่เพียง Variance ใน X\n"
   ]
  },
  {
   "id": "code-024",
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cross_decomposition import PLSRegression\n",
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "Xreg, yreg = make_regression(n_samples=800, n_features=60, n_informative=8,\n",
    "                             noise=25, random_state=SEED)\n",
    "Xrt, Xrv, yrt, yrv = train_test_split(Xreg, yreg, test_size=.25, random_state=SEED)\n",
    "pls = PLSRegression(n_components=8, scale=True, max_iter=1000)\n",
    "Zrt = pls.fit_transform(Xrt, yrt)[0]\n",
    "Zrv = pls.transform(Xrv)\n",
    "pred = pls.predict(Xrv).ravel()\n",
    "print(Zrt.shape, \"RMSE:\", mean_squared_error(yrv, pred)**.5)\n"
   ]
  },
  {
   "id": "md-025",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10 · Autoencoder — optional nonlinear bottleneck\n",
    "\n",
    "Cell นี้รันเมื่อมี PyTorch หากไม่มี package จะข้ามโดยไม่ทำให้ Notebook ส่วนอื่นเสีย\n"
   ]
  },
  {
   "id": "code-026",
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    import torch\n",
    "    from torch import nn\n",
    "    torch.manual_seed(SEED)\n",
    "    class AE(nn.Module):\n",
    "        def __init__(self, p, k=8):\n",
    "            super().__init__()\n",
    "            self.encoder = nn.Sequential(nn.Linear(p, 32), nn.ReLU(), nn.Linear(32, k))\n",
    "            self.decoder = nn.Sequential(nn.Linear(k, 32), nn.ReLU(), nn.Linear(32, p))\n",
    "        def forward(self, x):\n",
    "            z = self.encoder(x); return self.decoder(z), z\n",
    "    xt = torch.tensor(Xb_train, dtype=torch.float32)\n",
    "    xv = torch.tensor(Xb_test, dtype=torch.float32)\n",
    "    ae = AE(xt.shape[1], 8); opt = torch.optim.Adam(ae.parameters(), lr=1e-3)\n",
    "    for epoch in range(150):\n",
    "        xhat, _ = ae(xt); loss = ((xhat-xt)**2).mean()\n",
    "        opt.zero_grad(); loss.backward(); opt.step()\n",
    "    with torch.no_grad():\n",
    "        Za_train = ae.encoder(xt).numpy(); Za_test = ae.encoder(xv).numpy()\n",
    "    results.append(evaluate_representation(\"Autoencoder\", Za_train, Za_test))\n",
    "    print(\"latent\", Za_train.shape, \"train reconstruction\", float(loss))\n",
    "except ImportError:\n",
    "    print(\"PyTorch not installed — skip Autoencoder section\")\n"
   ]
  },
  {
   "id": "md-027",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11 · UMAP — optional fitted nonlinear mapper\n",
    "\n",
    "UMAP สามารถ transform new rows ด้วย fitted reducer แต่ต้อง version mapper และตรวจ distribution shift\n"
   ]
  },
  {
   "id": "code-028",
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    from umap import UMAP\n",
    "    umap_model = UMAP(n_neighbors=15, min_dist=.1, n_components=5,\n",
    "                      random_state=SEED, transform_seed=SEED)\n",
    "    Zu_train = umap_model.fit_transform(Xb_train)\n",
    "    Zu_test = umap_model.transform(Xb_test)\n",
    "    results.append(evaluate_representation(\"UMAP\", Zu_train, Zu_test))\n",
    "    print(Zu_train.shape, Zu_test.shape)\n",
    "except ImportError:\n",
    "    print(\"Install optional dependency with: pip install umap-learn\")\n"
   ]
  },
  {
   "id": "md-029",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 12 · t-SNE — visualization, not the default production feature\n",
    "\n",
    "ใช้ Digits dataset, ลดเป็น 30 PCA dimensions ก่อน แล้ววัด Trustworthiness\n",
    "\n",
    "> Standard scikit-learn t-SNE ไม่มี `transform(X_new)` จึงไม่ควรใช้พิกัดเป็น online feature โดยตรง\n"
   ]
  },
  {
   "id": "code-030",
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.manifold import TSNE, trustworthiness\n",
    "\n",
    "digits = load_digits()\n",
    "Xd = StandardScaler().fit_transform(digits.data)\n",
    "Xdp = PCA(n_components=30, random_state=SEED).fit_transform(Xd)\n",
    "tsne = TSNE(n_components=2, perplexity=30, init=\"pca\",\n",
    "            learning_rate=\"auto\", max_iter=1000, random_state=SEED)\n",
    "Zt = tsne.fit_transform(Xdp)\n",
    "print(\"trustworthiness:\", trustworthiness(Xdp, Zt, n_neighbors=10))\n",
    "plt.figure(figsize=(8,6)); plt.scatter(Zt[:,0], Zt[:,1], c=digits.target,\n",
    "    cmap=\"tab10\", s=8, alpha=.75); plt.colorbar(); plt.title(\"t-SNE of digits\"); plt.show()\n"
   ]
  },
  {
   "id": "md-031",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Experiment**\n",
    "1. เปลี่ยน perplexity เป็น 5, 15, 50\n",
    "2. เปลี่ยน random seed\n",
    "3. Cluster และช่องว่างเหมือนเดิมหรือไม่?\n",
    "4. สิ่งใดเป็นหลักฐาน Local Neighborhood และสิ่งใดเป็นเพียงรูปร่างของแผนที่?\n"
   ]
  },
  {
   "id": "md-032",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13 · Logistic score and log-odds as model-derived features\n",
    "\n",
    "Probability อยู่ในช่วง 0–1 ส่วน decision function ให้ log-odds ที่ไม่อิ่มตัวใกล้ขอบ\n"
   ]
  },
  {
   "id": "code-033",
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "score_pipe = Pipeline([\n",
    "    (\"impute\", SimpleImputer(strategy=\"median\")),\n",
    "    (\"scale\", StandardScaler()),\n",
    "    (\"logit\", LogisticRegression(max_iter=3000, random_state=SEED))\n",
    "])\n",
    "score_pipe.fit(X_train, y_train)\n",
    "test_logit = score_pipe.decision_function(X_test)\n",
    "test_prob = score_pipe.predict_proba(X_test)[:,1]\n",
    "pd.DataFrame({\"logit\":test_logit[:8], \"probability\":test_prob[:8]}).round(4)\n"
   ]
  },
  {
   "id": "md-034",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14 · Out-of-fold logistic features — prevent stacking leakage\n",
    "\n",
    "Training meta-feature ต้องมาจาก model ที่ไม่เคยเห็น label ของแถวนั้น\n"
   ]
  },
  {
   "id": "code-035",
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "base = Pipeline([\n",
    "    (\"impute\", SimpleImputer(strategy=\"median\")),\n",
    "    (\"scale\", StandardScaler()),\n",
    "    (\"logit\", LogisticRegression(max_iter=3000, random_state=SEED))\n",
    "])\n",
    "cv = StratifiedKFold(5, shuffle=True, random_state=SEED)\n",
    "oof_probability = cross_val_predict(base, X_train, y_train, cv=cv,\n",
    "                                    method=\"predict_proba\")[:,1]\n",
    "base.fit(X_train, y_train)\n",
    "future_probability = base.predict_proba(X_test)[:,1]\n",
    "print(oof_probability.shape, future_probability.shape,\n",
    "      roc_auc_score(y_train, oof_probability))\n"
   ]
  },
  {
   "id": "md-036",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 15 · Elastic-net logistic selection\n",
    "\n",
    "Non-zero coefficient เป็น model-specific shortlist ไม่ใช่ข้อพิสูจน์ว่า Feature อื่นไม่มีผล\n"
   ]
  },
  {
   "id": "code-037",
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "select_pipe = Pipeline([\n",
    "    (\"scale\", StandardScaler()),\n",
    "    (\"model\", LogisticRegression(penalty=\"elasticnet\", solver=\"saga\", max_iter=6000))\n",
    "])\n",
    "search = GridSearchCV(select_pipe, {\n",
    "    \"model__C\":[.01,.1,1], \"model__l1_ratio\":[.2,.5,.8]\n",
    "}, cv=5, scoring=\"roc_auc\", n_jobs=-1)\n",
    "search.fit(X_train, y_train)\n",
    "coef = search.best_estimator_[-1].coef_.ravel()\n",
    "selected = X.columns[coef != 0]\n",
    "print(search.best_params_, \"selected\", len(selected), \"of\", X.shape[1])\n",
    "list(selected[:10])\n"
   ]
  },
  {
   "id": "md-038",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 16 · Entity Embedding — optional PyTorch lookup\n",
    "\n",
    "ใช้ ID 0 เป็น unknown category และต้อง save ทั้ง mapping กับ weights\n"
   ]
  },
  {
   "id": "code-039",
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    import torch\n",
    "    from torch import nn\n",
    "    entity = nn.Embedding(num_embeddings=100_001, embedding_dim=16, padding_idx=0)\n",
    "    ids = torch.tensor([12, 980, 0], dtype=torch.long)\n",
    "    Ze = entity(ids)\n",
    "    numeric = torch.randn(3, 8)\n",
    "    downstream_input = torch.cat([Ze, numeric], dim=1)\n",
    "    print(\"embedding\", Ze.shape, \"combined\", downstream_input.shape)\n",
    "except ImportError:\n",
    "    print(\"PyTorch not installed — skip Entity Embedding section\")\n"
   ]
  },
  {
   "id": "md-040",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 17 · Compare representations\n",
    "\n",
    "ตารางนี้ไม่ได้ตัดสินผู้ชนะ เพราะ Hyperparameters และ Dataset ยังจำกัด แต่ช่วยให้นักศึกษาเห็น trade-off ระหว่าง Dimension, Performance และ Cost\n"
   ]
  },
  {
   "id": "code-041",
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comparison = pd.DataFrame(results).sort_values([\"auc\",\"log_loss\"], ascending=[False,True])\n",
    "comparison.style.format({\"auc\":\"{:.4f}\", \"log_loss\":\"{:.4f}\", \"model_seconds\":\"{:.4f}\"})\n"
   ]
  },
  {
   "id": "code-042",
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = comparison.sort_values(\"auc\").plot.barh(\n",
    "    x=\"representation\", y=\"auc\", figsize=(8,4), legend=False, color=\"#087e83\"\n",
    ")\n",
    "ax.set_xlim(max(.5, comparison.auc.min()-.03), 1); ax.set_xlabel(\"Test ROC AUC\")\n",
    "plt.tight_layout(); plt.show()\n"
   ]
  },
  {
   "id": "md-043",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 18 · Student exercises\n",
    "\n",
    "### Exercise A — honest model selection\n",
    "สร้าง nested cross-validation เพื่อเลือก PCA dimensions และ Logistic `C` พร้อมกัน\n",
    "\n",
    "### Exercise B — t-SNE interpretation audit\n",
    "รัน t-SNE อย่างน้อย 6 configurations แล้วบันทึกสิ่งที่คงอยู่/ไม่คงอยู่\n",
    "\n",
    "### Exercise C — representation contract\n",
    "สร้าง DataFrame ที่มี `entity_id`, `as_of`, `available_at`, `transform_version`, `component_id`, `value`\n",
    "\n",
    "### Exercise D — leakage demonstration\n",
    "เปรียบเทียบ in-sample logistic probability กับ OOF probability ใน meta-model\n",
    "\n",
    "### Exercise E — hybrid representation\n",
    "เปรียบเทียบ `Z only`, `raw only`, และ `[raw + Z]` โดยใช้ split เดียวกัน\n",
    "\n",
    "### Reflection\n",
    "Feature ใดหายไปหลัง Reduction และถ้า Feature นั้นสำคัญต่อ Safety/Fairness เราควรทำอย่างไร?\n"
   ]
  },
  {
   "id": "md-044",
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final checklist\n",
    "\n",
    "- [ ] Split ก่อน Fit ทุก Transformer\n",
    "- [ ] Supervised Feature ใช้ OOF/Cross-fitting\n",
    "- [ ] เก็บ fitted artifact, scaler, schema, seed และ version\n",
    "- [ ] Validate ทั้ง representation quality และ downstream task\n",
    "- [ ] ตรวจ drift ของ input และ latent features\n",
    "- [ ] อย่าอ่าน 2-D visualization เป็น global truth\n",
    "- [ ] เปรียบเทียบ raw, reduced และ hybrid baselines\n",
    "\n",
    "**Reduce dimensions, not meaning.**\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}