shortsmith: recursos medidos en el pod, no en el host

El primer render real murio OOMKilled con el limite de 1Gi que salia de medir en
la maquina de desarrollo. El host lleva ffmpeg 4.4.2 y la imagen 7.1.5, y 7.0
reescribio la transcodificacion sobre un planificador con colas de fotogramas
decodificados entre componentes: pasarle el WAV como segunda entrada dejaba al
decodificador de video correr por delante y llenar una cola con fotogramas de
6,2 MB. Arreglado partiendo el encode en dos pasos (repo shortsmith), y aqui van
las cifras nuevas, medidas dentro del pod.

requests 768Mi (pico anon 640 MB), limits 1536Mi. El limite no es 1Gi por el
caso techo de 180 s: ahi el pico era 940 MB de 1024 y sobrevivia solo porque
quedaba cache reclamable que soltar. A 1.5Gi memory.events da max 0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-01 16:40:24 +00:00
co-authored by Claude Opus 5
parent 0a6d0e9b58
commit aeb31eae59
+28 -23
View File
@@ -56,33 +56,38 @@ spec:
volumeMounts: volumeMounts:
- name: data - name: data
mountPath: /data mountPath: /data
# Measured on chemavx-k8 under `systemd-run --scope -p CPUQuota=300%`, # Measured IN THIS POD, sampling the cgroup's own memory.stat every 30 ms.
# sampling the cgroup's memory.stat anon every 20 ms — the figure that # An earlier set of numbers taken on the dev host said 522 MB and produced a
# cannot be reclaimed. Rendering is this service's normal mode, not a # 1Gi limit; the first real render was OOMKilled at 1483 MB. The host runs
# burst, so the request comes from the peak and not from the resting 22 MB. # ffmpeg 4.4.2 and the image ships 7.1.5, and 7.0 rewrote transcoding onto a
# threaded scheduler whose inter-component queues hold decoded frames. So:
# numbers measured outside the deployment target are a guess about it.
# #
# at rest (imports + FastAPI app) 22 MB # Rendering is this service's normal mode, not a burst, so the request comes
# frames, 3 workers, 1260 of them 147 MB # from the peak and not from the resting figure.
# audio synthesis and WAV 71 MB
# encode, -threads 3 480 MB <- the whole cost
# full render of examples/jal1628.json 522 MB, 32 s
# the same at the spec's 180 s ceiling 511 MB, 138 s
# #
# The peak is the encode's and does not grow with duration: at 5400 frames # at rest (imports + FastAPI app) 40 MB current
# anon is unchanged and only the page cache grows (64 -> 275 MB), which is # frames, 3 workers, 1260 of them +145 MB anon
# reclaimable. That is why these are sized against anon and not against # audio synthesis and WAV +82 MB anon
# memory.current, which reaches 808 MB at the ceiling. # encode pass 1, video only, -threads 3 +562 MB anon <- the cost
# encode pass 2, mux the audio in +15 MB anon
# two renders back to back, peak 640 MB anon / 716 current
# one render at the spec's 180 s ceiling 620 MB anon / 923 current
# #
# 512Mi does survive the reference render, but only by evicting page cache # anon is flat with duration; what grows is the page cache holding the PNG
# to sit exactly on the cap with no margin. 640Mi covers the measured peak # frames (70 MB at 42 s, 287 MB at the 180 s ceiling). That is reclaimable,
# with ~20% headroom; the 1Gi limit lets the page cache stay cached instead # so the request is sized against anon and the limit against current.
# of being re-read during the encode. #
# 768Mi covers the 640 MB anon peak with ~20% headroom. The limit is 1.5Gi
# and not 1Gi because of the ceiling case: at 1Gi that render peaked at
# 940 MB of 1024, i.e. 84 MB of margin, surviving only because the kernel
# had reclaimable cache to drop. Verified at 1.5Gi with memory.events
# reading `max 0` — the limit was never reached, not merely never fatal.
# #
# The CPU limit is what sizes the worker pool, not the request — the code # The CPU limit is what sizes the worker pool, not the request — the code
# reads cpu.max, since cpu_count() reports the node's 16. At limits.cpu 3 # reads cpu.max, since cpu_count() reports the node's 16. At limits.cpu 3
# that is 3 workers, verified in a real cgroup (300%->3, 200%->2, 100%->1). # that is 3 workers, verified in this pod. 42 s of video renders in ~31 s,
# 3 CPUs render in 32 s against 48 s at 2, for 22 MB less, so the third is # the 180 s ceiling in ~135 s.
# worth more than the extra worker costs.
# #
# Consequence, deliberate: under sustained contention the pod runs 3 workers # Consequence, deliberate: under sustained contention the pod runs 3 workers
# on the 1 CPU it is guaranteed and a render takes about 3x longer. That is # on the 1 CPU it is guaranteed and a render takes about 3x longer. That is
@@ -94,10 +99,10 @@ spec:
resources: resources:
requests: requests:
cpu: "1" cpu: "1"
memory: "640Mi" memory: "768Mi"
limits: limits:
cpu: "3" cpu: "3"
memory: "1Gi" memory: "1536Mi"
# The render runs off the event loop (asyncio.to_thread, and the frame # The render runs off the event loop (asyncio.to_thread, and the frame
# workers are separate processes), so /healthz answers while a job is in # workers are separate processes), so /healthz answers while a job is in
# flight. A probe timing out means the process is actually wedged. # flight. A probe timing out means the process is actually wedged.