Service Catalog¶

mkdir -p apps/base/myservice

kustomize build clusters/local/apps | kubectl apply -f -
# Test that it works

git add apps/
git commit -m "feat: add myservice"
git push
# Flux auto-deploys within 10 seconds

apps/base/myservice/
├── SERVICE.md                    # Documentation (START HERE)
├── kustomization.yaml            # Kustomize entry point
├── namespace.yaml                # Namespace definition
├── deployment.yaml               # Pod definition
├── service.yaml                  # Internal service
├── ingress.yaml                  # External access (if needed)
├── configmap.yaml                # Non-secret config
├── secret.yaml                   # Secret values (use SealedSecrets!)
├── pvc.yaml                      # Persistent storage (if needed)
├── hpa.yaml                      # Auto-scaling (if needed)
└── monitoring.yaml               # Prometheus scrape config (if applicable)

# My Service

## Overview
**Name**: myservice  
**Purpose**: What this service does  
**Owner**: Your name or team  
**Last Updated**: YYYY-MM-DD  
**Status**: ✅ Active / ⚠️ Testing / ❌ Deprecated

## Quick Facts

| Property | Value |
|----------|-------|
| **Image** | registry/myapp:1.0 |
| **Replicas** | 2 (minimum for HA) |
| **CPU Request** | 500m |
| **Memory Request** | 512Mi |
| **Storage** | 10Gi Longhorn |
| **Port** | 8080 |
| **URL** | https://myservice.yourdomain.com |

## Architecture

## Dependencies

### Required Services
- [ ] PostgreSQL (database)
- [ ] Redis (caching)
- [ ] Minio (object storage)

### Required Secrets
- [ ] `myservice-db-secret` (database password)
- [ ] `myservice-api-key` (external API key)

### Storage Requirements
- [ ] `/data` - PVC `myservice-data` (Longhorn, 10Gi, 2 replicas)
- [ ] `/tmp` - EmptyDir (ephemeral)

## Deployment

### Local Testing (k3d)
```bash
# Deploy to local cluster
kustomize build apps/local/myservice | kubectl apply -f -

# Wait for pod
kubectl wait --for=condition=ready pod -l app=myservice -n apps --timeout=120s

# Check logs
kubectl logs -f deployment/myservice -n apps

# Test connectivity
kubectl port-forward svc/myservice 8080:8080 -n apps
curl http://localhost:8080

# Git push triggers Flux auto-deployment
git add apps/base/myservice/
git commit -m "feat: add myservice"
git push

# Verify on production
kubectl -n apps get pods,svc

# Database backup (if applicable)
# Automated: CronJob daily at 2 AM
# Location: /backup/myservice_db_YYYYMMDD_HHMMSS.sql.gz

# Restore procedure
kubectl exec -it deployment/myservice-db -- psql -U admin -d mydb < backup.sql

kubectl describe pod <pod-name> -n apps
# Check: PVC status, resource requests, node availability

kubectl top pods -n apps
kubectl get pvc -n apps
# Scale up memory requests in deployment

kubectl logs -f deployment/myservice -n apps | grep -i database
# Verify: database pod running, credentials correct, network access

kubectl top pod <pod-name> -n apps
# Check: CPU/memory pressure, database queries, disk I/O

# 1. Test on k3d first
kustomize build apps/local/myservice | kubectl apply -f -

# 2. Update image tag in deployment.yaml
# 3. Commit to Git
git push

# 4. Flux auto-deploys with rolling update (zero downtime)
# 5. Monitor metrics during rollout
kubectl rollout status deployment/myservice -n apps

# Option 1: Revert Git commit
git revert <bad-commit-hash>
git push
# Flux auto-reverts deployment

# Option 2: Manual rollback
kubectl rollout undo deployment/myservice -n apps

# Nginx Web Server

## Overview
**Purpose**: Static website hosting  
**Status**: ✅ Active  

## Quick Facts
| Property | Value |
|----------|-------|
| **Image** | nginx:1.25-alpine |
| **Replicas** | 2 |
| **CPU** | 100m request, 500m limit |
| **Memory** | 64Mi request, 256Mi limit |
| **Storage** | 5Gi for website files |
| **URL** | https://mywebsite.com |

## Deployment
Simple: Just deploy from base manifest, no special requirements.

## Monitoring
- Monitor: HTTP response codes
- Alert: 5xx errors > 1%

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: apps
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.25-alpine
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 100m
            memory: 64Mi
          limits:
            cpu: 500m
            memory: 256Mi
        volumeMounts:
        - name: content
          mountPath: /usr/share/nginx/html
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 10
          periodSeconds: 10
      volumes:
      - name: content
        persistentVolumeClaim:
          claimName: nginx-content

# Redis Cache

## Overview
**Purpose**: In-memory data caching  
**Status**: ✅ Active  

## Quick Facts
| Property | Value |
|----------|-------|
| **Image** | redis:7-alpine |
| **Replicas** | 1 (stateful) |
| **CPU** | 200m |
| **Memory** | 512Mi request, 1Gi limit |
| **Storage** | 5Gi for persistence |
| **Port** | 6379 |

## Network
- **Service**: redis
- **DNS**: redis.apps.svc.cluster.local:6379

## Storage
- **RDB snapshots**: Persistent via Longhorn
- **Backup**: Nightly RDB export to backup storage

Deployment: {service-name}
Service: {service-name}
ConfigMap: {service-name}-config
Secret: {service-name}-secret
PVC: {service-name}-{data-type}  (e.g., redis-data, moodle-files)
Namespace: apps (default for all user services)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myservice
  namespace: apps
  annotations:
    traefik.ingress.kubernetes.io/router.middlewares: apps-sablier@kubernetescrd
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  ingressClassName: traefik
  tls:
  - hosts:
    - myservice.yourdomain.com
    secretName: myservice-tls

  rules:
  - host: myservice.yourdomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myservice
            port:
              number: 8080

kubectl port-forward svc/myservice 8080:8080 -n apps

Web Frontend (nginx)
    ↓
Backend API (myservice)
    ↓
PostgreSQL Database
    ↓
Longhorn Storage

Optional:
Redis ← Cache layer
Minio ← Object storage
Elasticsearch ← Logging

# Deploy a service
kustomize build apps/base/myservice | kubectl apply -f -

# Restart all pods for a service
kubectl rollout restart deployment/myservice -n apps

# Scale service
kubectl scale deployment/myservice --replicas=5 -n apps

# View logs
kubectl logs -f deployment/myservice -n apps

# Port-forward for local access
kubectl port-forward svc/myservice 8080:8080 -n apps

# Check resource usage
kubectl top pods -n apps
kubectl describe deployment/myservice -n apps

# Check storage
kubectl get pvc -n apps
kubectl describe pvc myservice-data -n apps

# Execute command in pod
kubectl exec -it pod/myservice-xyz-abc -- /bin/sh

# Delete service (including PVCs!)
kustomize build apps/base/myservice | kubectl delete -f -

CREATE
  ↓
DEPLOY (test on k3d first)
  ↓
PROMOTE (push to Git → Flux deploys to prod)
  ↓
MONITOR (watch metrics, logs)
  ↓
MAINTAIN (update image, patch vulnerabilities)
  ↓
UPGRADE (test locally, deploy via Git)
  ↓
DEPRECATE (mark as deprecated, set timeout)
  ↓
RETIRE (remove manifests, delete from production)

Internet
  ↓
Traefik Ingress (k3stat.serlo.lu)
  ↓
Skooner Service (ClusterIP)
  ↓
Skooner Pod
  ↓
Kubernetes API Server

# Check deployment status
kubectl get pods -n monitoring -l app=skooner
kubectl get svc -n monitoring -l app=skooner
kubectl get ingress -n monitoring -l app=skooner

# Check Flux HelmRelease
flux get helmrelease skooner -n monitoring

# Create token for Skooner service account (if exists)
kubectl create token skooner-sa -n monitoring --duration=24h

# If service account doesn't exist, use default
kubectl create token default -n monitoring --duration=24h

# Or use cluster-admin token
kubectl create token default -n kube-system --duration=24h

# Check pod status
kubectl get pods -n monitoring -l app=skooner
kubectl describe pod -l app=skooner -n monitoring

# Check service
kubectl get svc -n monitoring -l app=skooner
kubectl describe svc skooner -n monitoring

# Check ingress
kubectl get ingress -n monitoring -l app=skooner
kubectl describe ingress -n monitoring -l app=skooner

# Check logs
kubectl logs -f deployment/skooner -n monitoring

# Check HelmRelease status
flux get helmrelease skooner -n monitoring

# Check HelmRepository source
flux get source helm skooner -n flux-system

# Force reconciliation
flux reconcile helmrelease skooner -n monitoring
flux reconcile source helm skooner -n flux-system

# Check Flux logs
flux logs --follow helmrelease/skooner -n monitoring

# Verify service account exists
kubectl get sa -n monitoring

# Create service account if missing
kubectl create sa skooner-sa -n monitoring

# Grant permissions (adjust as needed)
kubectl create clusterrolebinding skooner-admin \
  --clusterrole=cluster-admin \
  --serviceaccount=monitoring:skooner-sa

# Check current chart version
flux get helmrelease skooner -n monitoring

# Update version in release.yaml (if pinned)
# Edit: infrastructure/base/skooner/release.yaml
# Commit and push - Flux will auto-update

kubectl rollout restart deployment/skooner -n monitoring

kubectl top pods -n monitoring -l app=skooner
kubectl describe pod -l app=skooner -n monitoring

values:
  service:
    type: ClusterIP
    port: 80
  ingress:
    enabled: true
    ingressClassName: traefik
    hosts:
      - host: k3stat.serlo.lu
        paths:
          - path: /
            pathType: ImplementationSpecific
  metrics:
    enabled: true