Hi! I would like to host a transparent proxy for cache.nixos.org on my local kubernetes cluster.

I took the following NGINX config https://nixos.wiki/wiki/FAQ/Private_Cache_Proxy and created all the folders on the mounted storage.

This is the kubernetes deployment:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nix-cache-volume
spec:
  capacity:
    storage: 500Gi
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/k8s/nix-cache" # Needs exists before PV is created!
  persistentVolumeReclaimPolicy: Retain
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nix-cache-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: manual
  resources:
    requests:
      storage: 500Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nix-cache
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nix-cache
  template:
    metadata:
      labels:
        app: nix-cache
        name: nix-cache
    spec:
      volumes:
        - name: nix-cache-storage
          persistentVolumeClaim:
            claimName: nix-cache-pvc
        - name: nix-cache-config
          configMap:
            name: nix-cache-config
      containers:
        - name: nix-cache
          image: nginx:1.27.0 
          ports:
            - containerPort: 80
          volumeMounts:
            - name: nix-cache-storage
              mountPath: /data
            - name: nix-cache-config
              mountPath: /etc/nginx/sites-available/default
          resources:
            limits:
              memory: "512Mi"
              cpu: "300m"
            requests:
              memory: "256Mi"
              cpu: "200m"
---
apiVersion: v1
kind: Service
metadata:
  name: nix-cache
spec:
  selector:
    app: nix-cache
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nix-cache-ingress
  annotations:
    traefik.ingress.kubernetes.io/router.tls: "true"
spec:
  rules:
    - host: "nix-cache.raspi.home"
      http:
        paths:
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: nix-cache
                port:
                  number: 80
  tls:
    - secretName: nix-cache-raspi-home-tls
      hosts:
        - "nix-cache.raspi.home"
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: nix-cache.raspi.home
spec:
  commonName: nix-cache.raspi.home
  dnsNames:
    - "nix-cache.raspi.home"
  secretName: nix-cache-raspi-home-tls
  issuerRef:
    name: ca-issuer
    kind: ClusterIssuer
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nix-cache-config
data:
  nginx.conf: |
    server {
      listen 80;
      server_name nix-cache.raspi.home;

      location ~ ^/nix-cache-info {
        proxy_store        on;
        proxy_store_access user:rw group:rw all:r;
        proxy_temp_path    /data/nginx/nix-cache-info/temp;
        root               /data/nginx/nix-cache-info/store;

        proxy_set_header Host "cache.nixos.org";
        proxy_pass https://cache.nixos.org;
      }

      location ~^/nar/.+$ {
        proxy_store        on;
        proxy_store_access user:rw group:rw all:r;
        proxy_temp_path    /data/nginx/nar/temp;
        root               /data/nginx/nar/store;

        proxy_set_header Host "cache.nixos.org";
        proxy_pass https://cache.nixos.org;
      }
    }

To use the cache I added it to the substituters.

  nix.settings.substituters = [
    "https://nix-cache.raspi.home/"
  ];

But when I try to use it, get the error:

# Trigger a download
nix develop nixpkgs#just
# Error message
warning: 'https://nix-cache.raspi.home' does not appear to be a binary cache

In the logs of the NGINX I see the following error:

2024/08/03 12:09:30 [error] 31#31: *3 open() "/usr/share/nginx/html/nix-cache-info" failed (2: No such file or directory), client: 10.42.2.7, server: localhost, request: "GET /nix-cache-info HTTP/1 │
│ 10.42.2.7 - - [03/Aug/2024:12:09:30 +0000] "GET /nix-cache-info HTTP/1.1" 404 153 "-" "curl/8.8.0 Nix/2.18.5" "10.42.2.1"                                                                             │
│ 10.42.2.7 - - [03/Aug/2024:12:09:30 +0000] "PUT /nix-cache-info HTTP/1.1" 405 157 "-" "curl/8.8.0 Nix/2.18.5" "10.42.2.1"    

Any ideas whats wrong? I’m neither an nix nor an nginx expert, so maybe it is something really simple but I cannot figure it out.

  • secana@programming.devOP
    link
    fedilink
    English
    arrow-up
    2
    ·
    3 months ago

    Thanks for the response. You are right, the config was at the wrong path. Unfortunately, the config itself does not work, too. After a bit of testing around this worked for me:

    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: nix-cache-volume
    spec:
      capacity:
        storage: 500Gi
      storageClassName: manual
      accessModes:
        - ReadWriteOnce
      hostPath:
        path: "/mnt/k8s/nix-cache" # Needs exists before PV is created!
      persistentVolumeReclaimPolicy: Retain
    ---
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: nix-cache-pvc
    spec:
      accessModes:
        - ReadWriteOnce
      storageClassName: manual
      resources:
        requests:
          storage: 500Gi
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nix-cache
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: nix-cache
      template:
        metadata:
          labels:
            app: nix-cache
            name: nix-cache
        spec:
          volumes:
            - name: nix-cache-storage
              persistentVolumeClaim:
                claimName: nix-cache-pvc
            - name: nix-cache-config
              configMap:
                name: nix-cache-config
          containers:
            - name: nix-cache
              image: nginx:1.27.0 
              ports:
                - containerPort: 80
              volumeMounts:
                - name: nix-cache-storage
                  mountPath: /data
                - name: nix-cache-config
                  mountPath: /etc/nginx/nginx.conf
                  subPath: nginx.conf
                  readOnly: true
              resources:
                limits:
                  memory: "512Mi"
                  cpu: "300m"
                requests:
                  memory: "256Mi"
                  cpu: "200m"
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: nix-cache
    spec:
      selector:
        app: nix-cache
      ports:
        - protocol: TCP
          port: 80
          targetPort: 80
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: nix-cache-ingress
      annotations:
        traefik.ingress.kubernetes.io/router.tls: "true"
    spec:
      rules:
        - host: "nix-cache.raspi.home"
          http:
            paths:
              - pathType: Prefix
                path: "/"
                backend:
                  service:
                    name: nix-cache
                    port:
                      number: 80
      tls:
        - secretName: nix-cache-raspi-home-tls
          hosts:
            - "nix-cache.raspi.home"
    ---
    apiVersion: cert-manager.io/v1
    kind: Certificate
    metadata:
      name: nix-cache.raspi.home
    spec:
      commonName: nix-cache.raspi.home
      dnsNames:
        - "nix-cache.raspi.home"
      secretName: nix-cache-raspi-home-tls
      issuerRef:
        name: ca-issuer
        kind: ClusterIssuer
    ---
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: nix-cache-config
    data:
      # See: https://www.channable.com/tech/setting-up-a-private-nix-cache-for-fun-and-profit
      nginx.conf: |
        events {
            worker_connections 1024;
        }
        http {
          proxy_cache_path /data/nginx/cache max_size=500G keys_zone=cache_zone:50m inactive=365d;
          proxy_cache cache_zone;
          proxy_cache_valid 200 365d;
          proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_504 http_403 http_404 http_429;
          proxy_ignore_headers X-Accel-Expires Expires Cache-Control Set-Cookie;
          proxy_cache_lock on;
    
          server {
              listen 80;
    
              server_name nix-cache.raspi.home;
    
              location /nix-cache-info {
                  return 200 "StoreDir: /nix/store\nWantMassQuery: 1\nPriority: 41\n";
              }
    
              location / {
                  proxy_set_header Host $proxy_host;
                  proxy_pass https://cache.nixos.org;
              }
          }
        }
    
    

    The config is an adaption from this blog post: https://www.channable.com/tech/setting-up-a-private-nix-cache-for-fun-and-profit