MariaDB 10.5.8 memory leak (MyISAM) on master but not slave

mariadbmemory

We've recently migrated a legacy application from MySQL 5.5 on Amazon Linux to MariaDB 10.5.8 (latest, from the official repos) on Amazon Linux 2. The "hardware" went from an m3.large instance to an m5.large instance (both 8GB RAM, 2 CPUs).

The application uses MyISAM tables for the vast majority of its 18G data set.

No MariaDB config options were changed in the migration; the application & config settings are identical to the old server (which happily ran for months/years without leaks)

After the upgrade we're seeing mariadbd RSS growing linearly until it gets OOM killed by the kernel. It takes about 48 hours from server start to reach that point:

ram usage

The top graphs show the master, and the bottom ones show the replication slave (which does not appear to exhibit this leak; or at least much, much more slowly). The master server answers queries from users. It's not a "high load" application by any means; and it runs happily on the modestly powered instances we've been using.

The data was transferred via a dump/restore to the new server; we did not attempt to copy and upgrade the MySQL data files.

Output from the master server mysqltuner, config & show global status follows:

Tuner

# perl mysqltuner.pl 
 >>  MySQLTuner 1.7.20 - Major Hayden <major@mhtx.net>

[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 10.5.8-MariaDB-log
[OK] Operating on 64-bit architecture
 
-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +Aria +CSV +InnoDB +MEMORY +MRG_MyISAM +MyISAM +PERFORMANCE_SCHEMA +SEQUENCE 
[--] Data in MyISAM tables: 17.1G (Tables: 169)
[--] Data in InnoDB tables: 3.5M (Tables: 10)
[--] Data in MEMORY tables: 0B (Tables: 5)
[OK] Total fragmented tables: 0
 
-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.
 
-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 1d 1h 24m 53s (29M q [325.984 qps], 1M conn, TX: 21G, RX: 7G)
[--] Reads / Writes: 59% / 41%
[--] Binary logging is enabled (GTID MODE: OFF)
[--] Physical Memory     : 7.5G
[--] Max MySQL memory    : 8.8G
[--] Other process memory: 0B
[--] Total buffers: 1.3G global + 50.7M per thread (150 max threads)
[--] P_S Max memory usage: 72B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 5.4G (72.66% of installed RAM)
[!!] Maximum possible memory usage: 8.8G (116.85% of installed RAM)
[!!] Overall possible memory usage with other process exceeded memory
[OK] Slow queries: 0% (11/29M)
[OK] Highest usage of available connections: 55% (83/150)
[OK] Aborted connections: 0.00%  (0/1810983)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[!!] Query cache may be disabled by default due to mutex contention.
[OK] Query cache efficiency: 43.7% (9M cached / 21M selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 245K sorts)
[!!] Joins performed without indexes: 513
[!!] Temporary tables created on disk: 72% (45K on disk / 62K total)
[OK] Thread cache hit rate: 99% (3K created / 1M connections)
[OK] Table cache hit rate: 37% (350 open / 928 opened)
[OK] table_definition_cache(400) is upper than number of tables(375)
[OK] Open file limit used: 1% (526/32K)
[OK] Table locks acquired immediately: 99% (10M immediate / 11M locks)
[OK] Binlog cache memory access: 100.00% (18402 Memory / 18402 Total)
 
-------- Performance schema ------------------------------------------------------------------------
[--] Memory used by P_S: 72B
[--] Sys schema isn't installed.
 
-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is enabled.
[--] Thread Pool Size: 2 thread(s).
[--] Using default value is good enough for your version (10.5.8-MariaDB-log)
 
-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 40.7% (437M used / 1B cache)
[OK] Key buffer size / total MyISAM indexes: 1.0G/4.0G
[OK] Read Key buffer hit rate: 100.0% (809M cached / 259K reads)
[!!] Write Key buffer hit rate: 77.1% (58M cached / 44M writes)
 
-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 16.0M/3.5M
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (200 %): 32.0M * 1/16.0M should be equal to 25%
[--] Number of InnoDB Buffer Pool Chunk : 1 for 1 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 99.90% (507276 hits/ 507766 total)
[!!] InnoDB Write Log efficiency: 7109.49% (54672 hits/ 769 total)
[OK] InnoDB log waits: 0.00% (0 waits / 55441 writes)
 
-------- AriaDB Metrics ----------------------------------------------------------------------------
[--] AriaDB is enabled.
[OK] Aria pagecache size / total Aria indexes: 128.0M/336.0K
[OK] Aria pagecache hit rate: 96.3% (1M cached / 45K reads)
 
-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.
 
-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.
 
-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.
 
-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] This server is acting as master for 2 server(s).
[--] Binlog format: ROW
[--] XA support enabled: ON
[--] Semi synchronous replication Master: OFF
[--] Semi synchronous replication Slave: OFF
[--] No replication setup for this server or replication not started.
 
-------- Recommendations ---------------------------------------------------------------------------
Variables to adjust:
  *** MySQL's maximum memory usage is dangerously high ***
  *** Add RAM before increasing MySQL buffer variables ***
    query_cache_size (=0)
    query_cache_type (=0)
    join_buffer_size (> 16.0M, or always use indexes with JOINs)
    tmp_table_size (> 128M)
    max_heap_table_size (> 128M)
    innodb_log_file_size should be (=4M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.

CONFIG

--port=3306 \
--slave-sql-verify-checksum=0 \
--user=wmysql \
--log-slave-updates=1 \
--slave-net-timeout=60 \
--wait_timeout=31536000 \
--interactive_timeout=31536000 \
--key-buffer-size=1024M \
--query_cache_size=64M \
--query-cache-limit=256000 \
--thread_cache_size=16 \
--table_open_cache=2000 \
--master-retry-count=120960 \
--max-connections=150  \
--max-allowed-packet=32M \
--open-files-limit=2048 \
--ft-min-word-len=2 \
--binlog-format=ROW \
--binlog_annotate_row_events=OFF \
--join_buffer_size=16M \
--sql-mode=NO_ENGINE_SUBSTITUTION \
--character-set-server=utf8 \
--collation-server=utf8_general_ci \
--performance_schema=1 \
--tmp_table_size=128M \
--max_heap_table_size=128M \
--sync-binlog=0 \
--sync-relay-log=0 \
--sync-relay-log-info=0 \
--sync-master-info=0 \
--innodb_buffer_pool_size=16M \
--innodb_log_file_size=32M \
--innodb-flush-log-at-trx-commit=2 \
--innodb-flush-neighbors=0 \
--innodb-flush-method=O_DIRECT \

SHOW GLOBAL STATUS

Variable_name   Value
Aborted_clients 15
Aborted_connects    0
Aborted_connects_preauth    0
Access_denied_errors    0
Acl_column_grants   0
Acl_database_grants 35
Acl_function_grants 0
Acl_procedure_grants    0
Acl_package_spec_grants 0
Acl_package_body_grants 0
Acl_proxy_users 2
Acl_role_grants 0
Acl_roles   0
Acl_table_grants    3
Acl_users   23
Aria_pagecache_blocks_not_flushed   0
Aria_pagecache_blocks_unused    15644
Aria_pagecache_blocks_used  15
Aria_pagecache_read_requests    1228921
Aria_pagecache_reads    45270
Aria_pagecache_write_requests   90307
Aria_pagecache_writes   90241
Aria_transaction_log_syncs  69
Binlog_commits  7261991
Binlog_group_commits    7261607
Binlog_group_commit_trigger_count   0
Binlog_group_commit_trigger_lock_wait   0
Binlog_group_commit_trigger_timeout 0
Binlog_snapshot_file    mysql-bin.000019
Binlog_snapshot_position    47873714
Binlog_bytes_written    4776990821
Binlog_cache_disk_use   0
Binlog_cache_use    18446
Binlog_stmt_cache_disk_use  10353
Binlog_stmt_cache_use   7243545
Busy_time   0.000000
Bytes_received  7982669592
Bytes_sent  22995831502
Column_compressions 0
Column_decompressions   0
Com_admin_commands  1545
Com_alter_db    0
Com_alter_db_upgrade    0
Com_alter_event 0
Com_alter_function  0
Com_alter_procedure 0
Com_alter_server    0
Com_alter_sequence  0
Com_alter_table 0
Com_alter_tablespace    0
Com_alter_user  0
Com_analyze 0
Com_assign_to_keycache  0
Com_backup  0
Com_backup_lock 0
Com_begin   0
Com_binlog  0
Com_call_procedure  0
Com_change_db   6026654
Com_change_master   1
Com_check   0
Com_checksum    0
Com_commit  0
Com_compound_sql    0
Com_create_db   0
Com_create_event    0
Com_create_function 0
Com_create_index    0
Com_create_package  0
Com_create_package_body 0
Com_create_procedure    0
Com_create_role 0
Com_create_sequence 0
Com_create_server   0
Com_create_table    26
Com_create_temporary_table  0
Com_create_trigger  0
Com_create_udf  0
Com_create_user 0
Com_create_view 0
Com_dealloc_sql 0
Com_delete  24955
Com_delete_multi    18
Com_do  0
Com_drop_db 0
Com_drop_event  0
Com_drop_function   0
Com_drop_index  0
Com_drop_procedure  0
Com_drop_package    0
Com_drop_package_body   0
Com_drop_role   0
Com_drop_server 0
Com_drop_sequence   0
Com_drop_table  3
Com_drop_temporary_table    0
Com_drop_trigger    0
Com_drop_user   0
Com_drop_view   0
Com_empty_query 0
Com_execute_immediate   0
Com_execute_sql 0
Com_flush   13
Com_get_diagnostics 0
Com_grant   0
Com_grant_role  0
Com_ha_close    0
Com_ha_open 0
Com_ha_read 0
Com_help    0
Com_insert  4002047
Com_insert_select   0
Com_install_plugin  0
Com_kill    5
Com_load    0
Com_lock_tables 3
Com_multi   0
Com_optimize    0
Com_preload_keys    0
Com_prepare_sql 0
Com_purge   0
Com_purge_before_date   9
Com_release_savepoint   0
Com_rename_table    0
Com_rename_user 0
Com_repair  0
Com_replace 11590
Com_replace_select  0
Com_reset   3
Com_resignal    0
Com_revoke  0
Com_revoke_all  0
Com_revoke_role 0
Com_rollback    0
Com_rollback_to_savepoint   0
Com_savepoint   0
Com_select  11954173
Com_set_option  1825129
Com_show_authors    0
Com_show_binlog_events  0
Com_show_binlogs    0
Com_show_charsets   0
Com_show_collations 0
Com_show_contributors   0
Com_show_create_db  0
Com_show_create_event   0
Com_show_create_func    0
Com_show_create_package 0
Com_show_create_package_body    0
Com_show_create_proc    0
Com_show_create_table   144
Com_show_create_trigger 24
Com_show_create_user    0
Com_show_databases  28
Com_show_engine_logs    0
Com_show_engine_mutex   0
Com_show_engine_status  33
Com_show_errors 0
Com_show_events 0
Com_show_explain    0
Com_show_fields 45938
Com_show_function_status    0
Com_show_generic    0
Com_show_grants 0
Com_show_keys   0
Com_show_binlog_status  0
Com_show_open_tables    4
Com_show_package_status 0
Com_show_package_body_status    0
Com_show_plugins    0
Com_show_privileges 0
Com_show_procedure_status   0
Com_show_processlist    45
Com_show_profile    0
Com_show_profiles   0
Com_show_relaylog_events    0
Com_show_slave_hosts    5
Com_show_slave_status   8
Com_show_status 1569
Com_show_storage_engines    6
Com_show_table_status   72
Com_show_tables 29
Com_show_triggers   0
Com_show_variables  41
Com_show_warnings   0
Com_shutdown    0
Com_signal  0
Com_start_all_slaves    0
Com_start_slave 0
Com_stmt_close  0
Com_stmt_execute    0
Com_stmt_fetch  0
Com_stmt_prepare    0
Com_stmt_reprepare  0
Com_stmt_reset  0
Com_stmt_send_long_data 0
Com_stop_all_slaves 0
Com_stop_slave  2
Com_truncate    0
Com_uninstall_plugin    0
Com_unlock_tables   3
Com_update  4186198
Com_update_multi    0
Com_xa_commit   0
Com_xa_end  0
Com_xa_prepare  0
Com_xa_recover  0
Com_xa_rollback 0
Com_xa_start    0
Compression OFF
Connection_errors_accept    0
Connection_errors_internal  0
Connection_errors_max_connections   0
Connection_errors_peer_address  0
Connection_errors_select    0
Connection_errors_tcpwrap   0
Connections 1815529
Cpu_time    0.000000
Created_tmp_disk_tables 45259
Created_tmp_files   14
Created_tmp_tables  62403
Delayed_errors  0
Delayed_insert_threads  1
Delayed_writes  16379090
Delete_scan 59
Empty_queries   2026190
Executed_events 0
Executed_triggers   55791
Feature_application_time_periods    0
Feature_check_constraint    2
Feature_custom_aggregate_functions  0
Feature_delay_key_write 0
Feature_dynamic_columns 0
Feature_fulltext    18
Feature_gis 0
Feature_insert_returning    0
Feature_invisible_columns   0
Feature_json    141092
Feature_locale  0
Feature_subquery    185
Feature_system_versioning   0
Feature_timezone    3
Feature_trigger 74
Feature_window_functions    0
Feature_xml 0
Handler_commit  7302836
Handler_delete  39212
Handler_discover    160
Handler_external_lock   0
Handler_icp_attempts    11736079
Handler_icp_match   7983082
Handler_mrr_init    0
Handler_mrr_key_refills 0
Handler_mrr_rowid_refills   0
Handler_prepare 36892
Handler_read_first  11129
Handler_read_key    237983167
Handler_read_last   0
Handler_read_next   318575614
Handler_read_prev   297124
Handler_read_retry  0
Handler_read_rnd    15437152
Handler_read_rnd_deleted    643
Handler_read_rnd_next   380643982
Handler_rollback    0
Handler_savepoint   0
Handler_savepoint_rollback  0
Handler_tmp_delete  0
Handler_tmp_update  49715399
Handler_tmp_write   21116241
Handler_update  9560539
Handler_write   16850465
Innodb_adaptive_hash_hash_searches  0
Innodb_adaptive_hash_non_hash_searches  103195
Innodb_background_log_sync  91679
Innodb_buffered_aio_submitted   1
Innodb_buffer_pool_dump_status  
Innodb_buffer_pool_load_status  Buffer pool(s) load completed at 201215 22:57:36
Innodb_buffer_pool_resize_status    
Innodb_buffer_pool_load_incomplete  OFF
Innodb_buffer_pool_pages_data   629
Innodb_buffer_pool_bytes_data   10305536
Innodb_buffer_pool_pages_dirty  274
Innodb_buffer_pool_bytes_dirty  4489216
Innodb_buffer_pool_pages_flushed    478
Innodb_buffer_pool_pages_free   379
Innodb_buffer_pool_pages_made_not_young 0
Innodb_buffer_pool_pages_made_young 0
Innodb_buffer_pool_pages_misc   0
Innodb_buffer_pool_pages_old    252
Innodb_buffer_pool_pages_total  1008
Innodb_buffer_pool_pages_lru_flushed    0
Innodb_buffer_pool_read_ahead_rnd   0
Innodb_buffer_pool_read_ahead   0
Innodb_buffer_pool_read_ahead_evicted   0
Innodb_buffer_pool_read_requests    508730
Innodb_buffer_pool_reads    490
Innodb_buffer_pool_wait_free    0
Innodb_buffer_pool_write_requests   132317
Innodb_checkpoint_age   3223475
Innodb_checkpoint_max_age   26469643
Innodb_data_fsyncs  18722
Innodb_data_pending_fsyncs  0
Innodb_data_pending_reads   0
Innodb_data_pending_writes  0
Innodb_data_read    8044544
Innodb_data_reads   517
Innodb_data_writes  56062
Innodb_data_written 7831749
Innodb_dblwr_pages_written  197
Innodb_dblwr_writes 9
Innodb_deadlocks    0
Innodb_history_list_length  11
Innodb_ibuf_discarded_delete_marks  0
Innodb_ibuf_discarded_deletes   0
Innodb_ibuf_discarded_inserts   0
Innodb_ibuf_free_list   0
Innodb_ibuf_merged_delete_marks 0
Innodb_ibuf_merged_deletes  0
Innodb_ibuf_merged_inserts  8
Innodb_ibuf_merges  1
Innodb_ibuf_segment_size    2
Innodb_ibuf_size    1
Innodb_log_waits    0
Innodb_log_write_requests   769
Innodb_log_writes   55573
Innodb_lsn_current  3860567584
Innodb_lsn_flushed  3860567584
Innodb_lsn_last_checkpoint  3857344109
Innodb_master_thread_active_loops   18381
Innodb_master_thread_idle_loops 73300
Innodb_max_trx_id   22692107
Innodb_mem_adaptive_hash    0
Innodb_mem_dictionary   162336
Innodb_os_log_fsyncs    18678
Innodb_os_log_pending_fsyncs    0
Innodb_os_log_pending_writes    0
Innodb_os_log_written   37634048
Innodb_page_size    16384
Innodb_pages_created    139
Innodb_pages_read   490
Innodb_pages_written    477
Innodb_row_lock_current_waits   0
Innodb_row_lock_time    0
Innodb_row_lock_time_avg    0
Innodb_row_lock_time_max    0
Innodb_row_lock_waits   0
Innodb_rows_deleted 46
Innodb_rows_inserted    81
Innodb_rows_read    93598
Innodb_rows_updated 18340
Innodb_system_rows_deleted  2
Innodb_system_rows_inserted 0
Innodb_system_rows_read 5
Innodb_system_rows_updated  0
Innodb_num_open_files   16
Innodb_truncated_status_writes  0
Innodb_available_undo_logs  128
Innodb_undo_truncations 0
Innodb_page_compression_saved   0
Innodb_num_index_pages_written  0
Innodb_num_non_index_pages_written  0
Innodb_num_pages_page_compressed    0
Innodb_num_page_compressed_trim_op  0
Innodb_num_pages_page_decompressed  0
Innodb_num_pages_page_compression_error 0
Innodb_num_pages_encrypted  0
Innodb_num_pages_decrypted  0
Innodb_have_lz4 OFF
Innodb_have_lzo OFF
Innodb_have_lzma    ON
Innodb_have_bzip2   OFF
Innodb_have_snappy  OFF
Innodb_have_punch_hole  ON
Innodb_defragment_compression_failures  0
Innodb_defragment_failures  0
Innodb_defragment_count 0
Innodb_instant_alter_column 0
Innodb_onlineddl_rowlog_rows    0
Innodb_onlineddl_rowlog_pct_used    0
Innodb_onlineddl_pct_progress   0
Innodb_secondary_index_triggered_cluster_reads  21438
Innodb_secondary_index_triggered_cluster_reads_avoided  0
Innodb_encryption_rotation_pages_read_from_cache    0
Innodb_encryption_rotation_pages_read_from_disk 0
Innodb_encryption_rotation_pages_modified   0
Innodb_encryption_rotation_pages_flushed    0
Innodb_encryption_rotation_estimated_iops   0
Innodb_encryption_key_rotation_list_length  0
Innodb_encryption_n_merge_blocks_encrypted  0
Innodb_encryption_n_merge_blocks_decrypted  0
Innodb_encryption_n_rowlog_blocks_encrypted 0
Innodb_encryption_n_rowlog_blocks_decrypted 0
Innodb_encryption_n_temp_blocks_encrypted   0
Innodb_encryption_n_temp_blocks_decrypted   0
Innodb_encryption_num_key_requests  0
Key_blocks_not_flushed  0
Key_blocks_unused   619823
Key_blocks_used 686117
Key_blocks_warm 17
Key_read_requests   811218000
Key_reads   259373
Key_write_requests  58441964
Key_writes  45075438
Last_query_cost 0.000000
Master_gtid_wait_count  0
Master_gtid_wait_time   0
Master_gtid_wait_timeouts   0
Max_statement_time_exceeded 0
Max_used_connections    83
Memory_used 346709208
Memory_used_initial 272257208
Not_flushed_delayed_rows    0
Open_files  526
Open_streams    4
Open_table_definitions  295
Open_tables 350
Opened_files    185495
Opened_plugin_libraries 0
Opened_table_definitions    663
Opened_tables   928
Opened_views    3135
Performance_schema_accounts_lost    0
Performance_schema_cond_classes_lost    0
Performance_schema_cond_instances_lost  0
Performance_schema_digest_lost  0
Performance_schema_file_classes_lost    0
Performance_schema_file_handles_lost    0
Performance_schema_file_instances_lost  0
Performance_schema_hosts_lost   0
Performance_schema_index_stat_lost  0
Performance_schema_locker_lost  0
Performance_schema_memory_classes_lost  0
Performance_schema_metadata_lock_lost   0
Performance_schema_mutex_classes_lost   0
Performance_schema_mutex_instances_lost 0
Performance_schema_nested_statement_lost    0
Performance_schema_prepared_statements_lost 0
Performance_schema_program_lost 0
Performance_schema_rwlock_classes_lost  0
Performance_schema_rwlock_instances_lost    0
Performance_schema_session_connect_attrs_lost   0
Performance_schema_socket_classes_lost  0
Performance_schema_socket_instances_lost    0
Performance_schema_stage_classes_lost   0
Performance_schema_statement_classes_lost   0
Performance_schema_table_handles_lost   0
Performance_schema_table_instances_lost 0
Performance_schema_table_lock_stat_lost 0
Performance_schema_thread_classes_lost  0
Performance_schema_thread_instances_lost    0
Performance_schema_users_lost   0
Prepared_stmt_count 0
Qcache_free_blocks  923
Qcache_free_memory  63638200
Qcache_hits 9275958
Qcache_inserts  2669842
Qcache_lowmem_prunes    0
Qcache_not_cached   7782
Qcache_queries_in_cache 1987
Qcache_total_blocks 4955
Queries 29949837
Questions   29893779
Rows_read   805140712
Rows_sent   66360601
Rows_tmp_read   81983757
Rpl_semi_sync_master_clients    0
Rpl_semi_sync_master_get_ack    0
Rpl_semi_sync_master_net_avg_wait_time  0
Rpl_semi_sync_master_net_wait_time  0
Rpl_semi_sync_master_net_waits  0
Rpl_semi_sync_master_no_times   0
Rpl_semi_sync_master_no_tx  0
Rpl_semi_sync_master_request_ack    0
Rpl_semi_sync_master_status OFF
Rpl_semi_sync_master_timefunc_failures  0
Rpl_semi_sync_master_tx_avg_wait_time   0
Rpl_semi_sync_master_tx_wait_time   0
Rpl_semi_sync_master_tx_waits   0
Rpl_semi_sync_master_wait_pos_backtraverse  0
Rpl_semi_sync_master_wait_sessions  0
Rpl_semi_sync_master_yes_tx 0
Rpl_semi_sync_slave_send_ack    0
Rpl_semi_sync_slave_status  OFF
Rpl_status  AUTH_MASTER
Rpl_transactions_multi_engine   0
Select_full_join    515
Select_full_range_join  77
Select_range    718961
Select_range_check  0
Select_scan 59837
Slave_connections   13
Slave_heartbeat_period  30.000
Slave_open_temp_tables  0
Slave_received_heartbeats   0
Slave_retried_transactions  0
Slave_running   OFF
Slave_skipped_errors    0
Slaves_connected    2
Slaves_running  0
Slow_launch_threads 0
Slow_queries    11
Sort_merge_passes   0
Sort_priority_queue_sorts   186158
Sort_range  242712
Sort_rows   16675942
Sort_scan   3692
Ssl_accept_renegotiates 0
Ssl_accepts 0
Ssl_callback_cache_hits 0
Ssl_cipher  
Ssl_cipher_list 
Ssl_client_connects 0
Ssl_connect_renegotiates    0
Ssl_ctx_verify_depth    0
Ssl_ctx_verify_mode 0
Ssl_default_timeout 0
Ssl_finished_accepts    0
Ssl_finished_connects   0
Ssl_server_not_after    
Ssl_server_not_before   
Ssl_session_cache_hits  0
Ssl_session_cache_misses    0
Ssl_session_cache_mode  NONE
Ssl_session_cache_overflows 0
Ssl_session_cache_size  0
Ssl_session_cache_timeouts  0
Ssl_sessions_reused 0
Ssl_used_session_cache_entries  0
Ssl_verify_depth    0
Ssl_verify_mode 0
Ssl_version 
Subquery_cache_hit  0
Subquery_cache_miss 138
Syncs   289
Table_locks_immediate   11018596
Table_locks_waited  31754
Table_open_cache_active_instances   1
Table_open_cache_hits   7609867
Table_open_cache_misses 11910
Table_open_cache_overflows  0
Tc_log_max_pages_used   0
Tc_log_page_size    0
Tc_log_page_waits   0
Threadpool_idle_threads 0
Threadpool_threads  0
Threads_cached  15
Threads_connected   48
Threads_created 3049
Threads_running 4
Transactions_gtid_foreign_engine    0
Transactions_multi_engine   0
Update_scan 1
Uptime  91711
Uptime_since_flush_status   91711
wsrep   0
wsrep_applier_thread_count  0
wsrep_cluster_capabilities  
wsrep_cluster_conf_id   18446744073709551615
wsrep_cluster_size  0
wsrep_cluster_state_uuid    
wsrep_cluster_status    Disconnected
wsrep_connected OFF
wsrep_local_bf_aborts   0
wsrep_local_index   18446744073709551615
wsrep_provider_capabilities 
wsrep_provider_name 
wsrep_provider_vendor   
wsrep_provider_version  
wsrep_ready OFF
wsrep_rollbacker_thread_count   0
wsrep_thread_count  0

Best Answer

After some work the the heap profiler I determined that our use INSERT DELAYED was causing the memory leak due to a MariaDB bug. Disabling use of INSERT DELAYED is an effective mitigation.