An incident report from the trenches of self-hosted infrastructure.
It was an ordinary day in the homelab.
Drupal 10 was running, MariaDB was minding its own business, Proxmox was doing its usual excellent job of keeping everything together, and I was busy doing what any responsible server administrator does: changing CSS until a logo finally decides to sit where it belongs.
Everything was fine.
Until it wasn't.
The symptoms
The MariaDB container was running out of memory. CPU wait was higher than expected. There was some CPU activity, memory pressure, and the database was apparently having a rather difficult afternoon.
Naturally, the first suspect was MariaDB.
After all, who else could possibly be responsible for a database container running out of memory?
I had recently been clearing Drupal caches quite a lot, rebuilding things, testing layouts, and generally making Drupal reconsider its life choices.
The obvious conclusion was that all this cache clearing was putting pressure on MariaDB.
So, being a responsible administrator, I increased the container memory limit from 1 GB to 2 GB.
Problem solved? Perhaps.
But something didn't quite add up.
The investigation
Back on the Proxmox host:
root@pxboss:~# dmesg -T | grep -i -E 'oom|killed process|out of memory'
And there it was.
[Wed Nov 4 19:01:34 2026] kworker/u14:0 invoked oom-killer
oom-kill:constraint=CONSTRAINT_MEMCG,
oom_memcg=/lxc/109,
task=nano,
pid=2289064,
uid=100000
Memory cgroup out of memory:
Killed process 2289064 (nano)
total-vm:809208kB
anon-rss:790400kB
Wait.
NANO?
Not MariaDB?
Not Apache?
Not PHP?
Not Drupal rebuilding its entire universe?
NANO?!
The plot twist
And then I remembered.
Yesterday, I had tried to open a rather large file.
A 700 MB file.
With Nano.
The attempt hadn't succeeded, so I simply moved on to other things.
Apparently, Nano had other plans.
Somewhere inside the MariaDB container, our little text editor was still sitting there, desperately trying to fulfil its master's request.
I imagine the conversation went something like this:
NANO: Master has requested a file.
NANO: The file is 700 MB.
NANO: This is a large file.
NANO: I must open it.
NANO: I must satisfy the master.
NANO: Allocate memory.
NANO: MORE MEMORY.
NANO: I CAN DO THIS.
Meanwhile, MariaDB:
MariaDB: Excuse me, why is memory disappearing?
MariaDB: I am just serving queries.
MariaDB: I have a 128 MB InnoDB buffer pool.
MariaDB: I am not even using that much memory.
MariaDB: Is somebody eating the RAM?
And Nano, now approaching 800 MB of resident memory:
NANO: I HAVE LOADED THE FILE.
NANO: I THINK.
NANO: I CANNOT SEE THE END.
NANO: THE END IS VERY FAR AWAY.
NANO: MASTER, I REQUIRE MORE MEMORY.
The database examination
Just to make sure MariaDB wasn't hiding anything, I checked:
MariaDB [(none)]> SHOW VARIABLES LIKE 'innodb_buffer_pool_size';
+-------------------------+-----------+
| Variable_name | Value |
+-------------------------+-----------+
| innodb_buffer_pool_size | 134217728 |
+-------------------------+-----------+
1 row in set (0.000 sec)
128 MB.
Then:
MariaDB [(none)]> SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_reads';
+--------------------------+-------+
| Variable_name | Value |
+--------------------------+-------+
| Innodb_buffer_pool_reads | 5557 |
+--------------------------+-------+
1 row in set (0.000 sec)
And finally:
MariaDB [(none)]> SHOW GLOBAL STATUS LIKE 'Created_tmp_disk_tables';
+-------------------------+-------+
| Variable_name | Value |
+-------------------------+-------+
| Created_tmp_disk_tables | 0 |
+-------------------------+-------+
MariaDB was practically sitting there with its hands in its pockets.
It was Nano who had been consuming almost 800 MB.
The aftermath
The OOM killer eventually lost patience and terminated Nano.
Killed process 2289064 (nano)
anon-rss:790400kB
And just like that, the database was innocent.
The 1 GB memory limit had not been defeated by Drupal, MariaDB, PHP, or some mysterious background process.
It had been defeated by a text editor trying to read a file that had absolutely no business being opened with a text editor in the first place.
I had increased the container limit to 2 GB, investigated database performance, checked InnoDB counters, and briefly suspected Drupal's cache rebuilding.
All because I forgot about Nano.
Lessons learned
- Never underestimate a text editor.
- A 700 MB file is not necessarily a good candidate for Nano.
- Before tuning MariaDB, check whether something else is eating all the RAM.
- The OOM killer doesn't care about your debugging session.
- Sometimes the database is innocent.
- Sometimes the administrator is the problem.
Epilogue: The next benchmark
I was actually planning to retry the performance comparison between Drupal 7 and Drupal 10.
You know, proper benchmarks. Same hardware, comparable content, database queries, response times, the whole thing.
But perhaps we should establish one additional benchmark condition:
root@pxboss:~# ps aux --sort=-rss | head
USER PID %CPU %MEM COMMAND
mysql 123 12.0 8.0 mariadbd
www-data 456 4.0 3.0 apache2
root 789 0.0 78.0 nano
If Nano is running, the benchmark is invalid.
If Nano is not running, we may proceed.
And if Nano is running with a 700 MB file open, we don't benchmark anything.
We simply increase the RAM and hope for the best.
Final diagnosis: PEBKAC (Problem Exists Between Keyboard And Chair).
No databases were harmed intentionally during this incident. One text editor was terminated in the line of duty.
Incident classified as: Unintentional memory stress test / Human-induced infrastructure malfunction.
- Log in to post comments
- 6 views