For future reference, I ran into a similar problem; I was creating too many file descriptors (FDs) by creating too many files and sockets (on Unix OSs, everything is a FD). My solution was to increase FDs at runtime with setrlimit().

First I got the FD limits, with the following code:

// This goes somewhere in your code
struct rlimit rlim;

if (getrlimit(RLIMIT_NOFILE, &rlim) == 0) {
    std::cout << "Soft limit: " << rlim.rlim_cur << std::endl;
    std::cout << "Hard limit: " << rlim.rlim_max << std::endl;
} else {
    std::cout << "Unable to get file descriptor limits" << std::endl;
}

After running getrlimit(), I could confirm that on my system, the soft limit is 256 FDs, and the hard limit is infinite FDs (this is different depending on your distro and specs). Since I was creating > 300 FDs between files and sockets, my code was crashing.

In my case I couldn’t decrease the number of FDs, so I decided to increase the FD soft limit instead, with this code:

// This goes somewhere in your code
struct rlimit rlim;

rlim.rlim_cur = NEW_SOFT_LIMIT;
rlim.rlim_max = NEW_HARD_LIMIT;

if (setrlimit(RLIMIT_NOFILE, &rlim) == -1) {
    std::cout << "Unable to set file descriptor limits" << std::endl;
}

Note that you can also get the number of FDs that you are using, and the source of these FDs, with this code.

Also you can find more information on gettrlimit() and setrlimit() here and here.

Очень часто при работе на высоконагруженных Linux серверах могут возникать ошибки “too many open files”. Это означает, что процесс открыл слишком много файлов (читай файловых дескрипторов) и не может открыть новые. В Linux ограничения “max open file limit“ установлены по умолчанию для каждого процесса и пользователя, и они не слишком высокие.

В данной статье мы рассмотрим, как проверить текущие ограничения на количество открытых файлов в Linux, как изменить этот параметр для всего сервера, для отдельных сервисов и для сеанса. Статья применима для большинства современных дистрибутивов Linux (Debian, Ubuntu, CentOS, RHEL, Oracle Linux, Rocky и т.д.)

Содержание:

  • Ошибка: Too many open files и лимиты на количество открытых файлов в Linux
  • Глобальные ограничения на количество открытых файлов в Linux
  • Увеличить лимит открытых файловых дескрипторов для отдельного сервиса
  • Увеличить количество открытых файлов для Nginx и Apache
  • Лимиты file-max для текущей сессии

Ошибка: Too many open files и лимиты на количество открытых файлов в Linux

Чаще всего ошибку “too many open files“. Чаще всего эта ошибка встречается на серверах с установленным веб-сервером NGINX/httpd, сервером БД (MySQL/MariaDB/PostgreSQL), при чтении большого количества логов. Например, когда веб-серверу Nginx не хватает лимита для открытия файлов, вы получите ошибку:

socket () failed (24: Too many open files) while connecting to upstream

ошибка too many opne files в nginx

Или:

HTTP: Accept error: accept tcp [::]:<port_number>: accept4: too many open files.

В Python:

OSError: [Errno 24] Too many open files.

Максимально количество файловых дескрипторов, которые могут быть открыты в вашей файловой системе всеми процессами можно узнать так:

# cat /proc/sys/fs/file-max

Чтобы узнать, сколько файлов открыто сейчас, выполните:

$ cat /proc/sys/fs/file-nr

7744    521      92233720
  • 7744 — суммарное количество открытых файлов
  • 521– число открытых файлов, но которые сейчас не используются
  • 92233720– максимальное количество файлов, которое разрешено открывать

В Linux ограничение на максимальное количество открытых файлов можно натсроить на нескольких уровнях:

  • Ядра ОС
  • Сервиса
  • Пользователя

Чтобы вывести текущее ограничение на количество открытых файлов в ядре Linux, выполните:

# sysctl fs.file-max

fs.file-max = 92233720

fs.file-max - максимальное количество открытых файлов в linux

Выведем ограничение на количество открытых файлов для одного процесса текущего пользователя:

# ulimit -n

По умолчанию количество файлов для одного процесса этого ограничено числом 1024.

Выведем максимальное количество для одного пользователя (max user processes):

# ulimit –u

5041

Умножаем 1024 * 5041 дает нам 5161984 – это максимально количество открытых файлов всеми процессами пользователя.

В Linux есть два типа ограничений на количество открытых файлов: Hard и Soft. Soft ограничения носят рекомендательный характер, при превышении значения пользователь будет получать предупреждения. Если количество открытых файлов превысило hard лимит, пользователь не сможет открыть новые файлы пока не будет закрыты ранее открытые.

Для просмотра текущих лимитов используется команда ulimit с опцией
-S
(soft) или
-H
(hard) и параметром
-n
(the maximum number of open file descriptors).

Для вывода Soft-ограничения:

# ulimit -Sn

Для вывода Hard-ограничения:

# ulimit -Hn

ulimit hard квота на количество открытых дескрипторов

Глобальные ограничения на количество открытых файлов в Linux

Чтобы разрешить всем сервисам открывать большее количество файлов, можно изменить лимиты на уровне всей ОС Linux. Чтобы новые настройки работали постоянно и не сбрасывались при перезапуске сервера или сессии, нужно поправить файл /etc/security/limits.conf. Строки с разрешениями выглядит так:

Имя_пользователя тип_огрничение название_ограничения значение

Например:

apache  hard nofile 978160
apache soft nofile 978160

Вместо имени пользователя можно указать
*
. Это означает, что это ограничение будет применяться для всех пользователей Linux:

* hard nofile 97816
* soft nofile 97816

Например, вы получили ошибку too many open files для nginx. Проверьте, сколько файлов разрешено открывать процессу этому пользователю:

$ sudo -u nginx bash -c 'ulimit -n'

1024

Для нагруженного сервера этого недостаточно. Добавьте в /etc/security/limits.conf строки

nginx hard nofile 50000
nginx soft nofile 50000

В старых ядрах Linux значение fs.file-max может быть равно 10000. Поэтому проверьте это значение и увеличьте его, чтобы оно было больше чем ограничение в limits.conf:

# sysctl -w fs.file-max=500000

Это временно увеличит лимит. Чтобы новые настройки стали постоянными, нужно добавить в файл /etc/sysctl.conf строку:

fs.file-max = 500000

Проверьте, что в файле /etc/pam.d/common-session (Debian/Ubuntu или /etc/pam.d/login для CentOS/RedHat/Fedora) есть строчка:

session required pam_limits.so

Если нет, добавьте ее в конец. Она нужна, чтобы ограничения загружались при авторизации пользователя.

После изменений, перезапустите терминал и проверьте значение лимита max_open_files:

# ulimit -n

97816

Увеличить лимит открытых файловых дескрипторов для отдельного сервиса

Вы можете изменить лимит на количество открытых файловых дескрипторов для конкретного сервиса, а не для всей системы. Рассмотрим на примере apache. Чтобы изменить значения, откройте настройки службы через systemctl:

# systemctl edit httpd.service

Добавьте необходимые лимиты, например:

[Service]
LimitNOFILE=16000
LimitNOFILESoft=16000

nginx.conf настроить limitnofile

После изменения, нужно обновить конфигурацию сервиса и перезапустить его:

# systemctl daemon-reload
# systemctl restart httpd.service

Чтобы проверить, изменились ли значения, нужно получить PID сервиса:

# systemctl status httpd.service

Например, вы определил PID сервиса 32724:

# cat /proc/32724/limits | grep "Max open files”

Значение должно быть 16000.

настройка max open filex для сервиса в linux centos

Так вы изменили значения Max open files для конкретного сервиса.

Увеличить количество открытых файлов для Nginx и Apache

После того, как вы увеличил ограничения на количество открытых файлов для сервера, нужно также поправить конфигурационный файл службы. Например, для веб-сервера Nginx в файле конфигурации /etc/nginx/nginx.conf нужно задать значение в директиве:

worker_rlimit_nofile 16000

Директива worker_rlimit_nofile задает ограничение на количество файлов, открытых в рабочем процессе (
RLIMIT_NOFILE
). В Nginx файловые дескрипторы нужны для возврата статического файла из кэша для каждого подключения клиента. Чем больше пользователей использует ваш сервер и чем больше статических файлов отдает nginx, тем большее количество дескрипторов используется. Сверху максимальное количество дескрипторов ограничивается на уровне ОС и/или сервиса. При превышении количеств открытых файлов nginx появится ошибка
socket() failed (24: Too many open files) while connecting to upstream
.

При настройке Nginx на высоконагруженном 8-ядерном сервере с worker_connections 8192 нужно в worker_rlimit_nofile указать 8192*2*8 (vCPU) = 131072.

После чего выполнить рестарт Nginx:

# nginx -t && service nginx -s reload

Чтобы увидеть чисто открытых файлов для процессов пользователя nginx:

# su nginx
# ulimit –Hn
# for pid in `pidof nginx`; do echo "$(< /proc/$pid/cmdline)"; egrep 'files|Limit' /proc/$pid/limits; echo "Currently open files: $(ls -1 /proc/$pid/fd | wc -l)"; echo; done

Для apache, нужно создать директорию:

# mkdir /lib/systemd/system/httpd.service.d/

После этого создайте файл limit_nofile.conf:

# nano /lib/systemd/system/httpd.service.d/limit_nofile.conf

httpd изменить LimitNOFILE

И добавьте в него:

[Service]
LimitNOFILE=16000

Не забудьте перезапустить сервис httpd.

Лимиты file-max для текущей сессии

Чтобы изменить лимиты на открытые файлы в рамках текущей сессии пользователя, выполните команду:

# ulimit -n 3000

Если указать здесь значение большее, чем заданное в hard limit, появится ошибка:

-bash: ulimit: open files: cannot modify limit: Operation not permitted

задать ulimit max_files для пользователя в linux

Когда вы завершите текущую сессию терминала и откроете новую, лимиты вернутся к начальным значениям, указанным в файле /etc/security/limits.conf.

В данной статье мы разобрались, как решить проблему с недостаточным лимитом для открытых файловых дескрипторов в Linux и рассмотрели несколько вариантов изменения лимитов на сервере.

Very often ‘too many open files’ errors occur on high-load Linux servers. It means that a process has opened too many files (file descriptors) and cannot open new ones. On Linux, the “max open file limit” is set by default per process or user and the values are rather small.

In this article, we’ll look at how to check the current limits on the maximum number of open files in Linux, and how to change this setting globally for the entire server, for specific services, and for a user session. The article is applicable to most modern Linux distros (Debian, Ubuntu, CentOS, RHEL, Oracle Linux, Rocky, etc.)

Contents:

  • ‘Too Many Open Files’ Error and Open Files Limit in Linux
  • How to Increase the Max Open Files Limit in Linux?
  • Increase the Maximum Number of Open File Descriptors per Service
  • How to Set Max Open Files for Nginx and Apache?
  • Change the Open File Limit for the Current User Session

‘Too Many Open Files’ Error and Open Files Limit in Linux

First of all, let’s see where the ‘too many open files’ errors appear. Most often it occurs on the servers with an installed Nginx /httpd web server or a database server running MySQL/MariaDB/PostgreSQL when reading a large number of log files. For example, when an Nginx exceeds the open files limit, you will see an error:

socket () failed (29: Too many open files) while connecting to upstream

Too many open files error in Linux log files

Or:

HTTP: Accept error: accept tcp [::]:<port_number>: accept4: too many open files.

In Python apps:

OSError: [Errno 24] Too many open files.

Using this command, you can get the maximum number of file descriptors your system can open:

# cat /proc/sys/fs/file-max

To find out how many files are currently open, run:

# cat /proc/sys/fs/file-nr

7122    123      92312720
  • 7122 — total number of open files
  • 123– number of open files that are not currently in use
  • 92312720– maximum number of files allowed to be opened

In Linux, you can configure max open files limits at several levels:

  • OS kernel
  • Service
  • User

To display the current limit on the number of open files in the Linux kernel, run:
# sysctl fs.file-max

fs.file-max = 92233720

fs.file-max in linux kernel

Let’s display the open files limit for one process of the current user:

# ulimit -n

By default, the number of files for one process of this is limited to 1024.

Let’s display the maximum number for one user (max user processes):

# ulimit –u

5041

Multiplying 1024 * 5041 gives us 5161984 – this is the maximum number of open files by all user processes.

There are two types of limits on the number of open files: Hard and Soft. Soft limits are advisory. If the number of open files has exceeded the hard limit, the user won’t be able to open new files until the previously opened ones are closed.

To view the current limits, use the ulimit command with the -S (soft) or -H (hard) option and the -n (the maximum number of open file descriptors) option.

To display the soft limit, run this command:

# ulimit –Sn

To display the hard limit value:

# ulimit -Hn
linux: ulimit hard max open files

How to Increase the Max Open Files Limit in Linux?

To allow all services to open a large number of files, you can change the limits in your Linux OS. To make new settings permanent and prevent their reset after a server or session restart, you must make changes to /etc/security/limits.conf. This file allows limiting the number of various system resources available to a user process. Use the following format:

username restriction_type restriction_name value

For example:

apache  hard nofile 978160
apache soft nofile 978160

You can use * instead of a username. This means that this open files limit will apply to all Linux users:

* hard nofile 97816
* soft nofile 97816

For example, you got a too many open files error for Nginx. Check how many files this user process is allowed to open:

$ sudo -u nginx bash -c 'ulimit -n'

1024

This is not enough for a high-load server. Add the following lines to /etc/security/limits.conf:

nginx hard nofile 50000
nginx soft nofile 50000

On older Linux kernels, the value of fs.file-max may be set to 10000. So check this value and increase it so that it is greater than the number in limits.conf:

# sysctl -w fs.file-max=500000

This will temporarily increase the open file limit. To make the new settings permanent, you need to add the following line to the /etc/sysctl.conf file:

fs.file-max = 500000

And apply it:

# sysctl -p

Check that the file /etc/pam.d/common-session (Debian/Ubuntu) or /etc/pam.d/login (CentOS/RedHat/Fedora) contains the line:

session required pam_limits.so

If not, add it to the end of the config file. This parameter allows applying open file limits after user authentication.

After making any changes, re-open the console, and check the max_open_files value:

# ulimit -n

50000

Increase the Maximum Number of Open File Descriptors per Service

You can increase the max open file descriptors for a specific service, rather than for the entire operating system. Let’s take apache as an example. Open the service settings using systemctl:

# systemctl edit httpd.service

Add the limits you want, e.g.:

[Service]
LimitNOFILE=16000
LimitNOFILESoft=16000

set LimitNOFILE for systemd service

After making the changes, update the service configuration, and restart it:

# systemctl daemon-reload
# systemctl restart httpd.service

To check if the values have changed, get the service PID:

# systemctl status httpd.service

For example, the service PID is 3724:

# cat /proc/3724/limits | grep "Max open files"

The value must be 16000.

get Max open files per process by PID

Thus, you have changed Max open files value for a specific service.

How to Set Max Open Files for Nginx and Apache?

After you have increased the limit on the number of open files for a server, you also have to change the service configuration file. For example, specify/change the following directive value in the Nginx configuration file /etc/nginx/nginx.conf:

worker_rlimit_nofile 16000

The worker_rlimit_nofile directive sets the limit on the number of files open by a worker process ( RLIMIT_NOFILE ). Nginx needs file descriptors to return a static file from the cache for each client connection. The more users connect to your server and the more static files Nginx returns, the more file descriptors are used. The maximum number of handles is limited at the OS and/or service level. If the number of open files in Nginx is exceeded, a “socket() failed (24: Too many open files) while connecting to upstream error” will occur.

[alert]When configuring Nginx on a high load 8-core server with worker_connections 8192, you need to specify 8192*2*8 (vCPU) = 131072 in worker_rlimit_nofile.

Then restart Nginx.

# nginx -t && service nginx -s reload

To see the number of open files for Nginx user processes:

# su nginx
# ulimit –Hn
# for pid in `pidof nginx`; do echo "$(< /proc/$pid/cmdline)"; egrep 'files|Limit' /proc/$pid/limits; echo "Currently open files: $(ls -1 /proc/$pid/fd | wc -l)"; echo; done

For Apache, you need to create a directory:

# mkdir /lib/systemd/system/httpd.service.d/

Then create the limit_nofile.conf file:

# nano /lib/systemd/system/httpd.service.d/limit_nofile.conf

LimitNOFILE set ulimiys for apache and nginx

Add to it:

[Service]
LimitNOFILE=16000

Don’t forget to restart the httpd service.

Change the Open File Limit for the Current User Session

To change the maximum open file limits for your current user session, run this command:

# ulimit -n 3000

If you specify a value here greater than that specified in the hard limit, an error will appear:

-bash: ulimit: open files: cannot modify limit: Operation not permitted

ulimit: open files user session

After closing the session and opening a new one, the limits will return to the initial values specified in /etc/security/limits.conf.

In this article, we have learned how to solve the issue when the value of open file descriptors limit in Linux is too small, and looked at several options for changing these limits on the server.

Comments

@SynclabIO

Checklist

  • The bug is reproducible against the latest release and/or master.
  • There are no similar issues or pull requests to fix it yet.

Describe the bug

Server will hit some limit (I don’t know what it is) and stop accept any new request.
Looks like Some connection is always alive

To reproduce

Host a server and keep making request on it after around 24H.

Expected behavior

The connection should be close after 60 sec.

Actual behavior

Get asyncio:socket.accept() out of system resource and OSError: [Errno 24] Too many open files error log when try to make new request on server.
I already make sure everything or browser is closed. But looks like some connection is still alive.

Debugging material

The ESTABLISHED will increase by time.

run lsof in container:

COMMAND  PID TID TASKCMD USER   FD      TYPE             DEVICE SIZE/OFF     NODE NAME
uvicorn    1             root    0u      CHR                1,3      0t0        6 /dev/null
uvicorn    1             root    1w     FIFO               0,13      0t0 18371653 pipe
uvicorn    1             root    2w     FIFO               0,13      0t0 18371654 pipe
uvicorn    1             root    3u  a_inode               0,14        0    13401 [eventpoll]
uvicorn    1             root    4u     unix 0x0000000000000000      0t0 18369360 type=STREAM
uvicorn    1             root    5u     unix 0x0000000000000000      0t0 18369361 type=STREAM
uvicorn    1             root    6u     IPv4           18369363      0t0      TCP *:https (LISTEN)
uvicorn    1             root    7u     IPv4           18413076      0t0      TCP 59c08b6aac89:https->61-66-209-161.askey.com.tw:27651 (ESTABLISHED)
uvicorn    1             root    8u     IPv4           18369366      0t0      TCP 59c08b6aac89:33846->ip-172-31-28-203.ap-northeast-1.compute.internal:postgresql (ESTABLISHED)
uvicorn    1             root    9u     IPv4           18600291      0t0      TCP 59c08b6aac89:https->125-227-151-121.HINET-IP.hinet.net:61830 (ESTABLISHED)
uvicorn    1             root   10u     IPv4           18384947      0t0      TCP 59c08b6aac89:https->61-222-56-55.HINET-IP.hinet.net:14377 (ESTABLISHED)
uvicorn    1             root   11u     IPv4           18388349      0t0      TCP 59c08b6aac89:https->210.241.98.253:12903 (ESTABLISHED)
uvicorn    1             root   12u     IPv4           18402240      0t0      TCP 59c08b6aac89:https->125-227-151-121.HINET-IP.hinet.net:57692 (ESTABLISHED)
uvicorn    1             root   13u     IPv4           18370708      0t0      TCP 59c08b6aac89:33856->ip-172-31-28-203.ap-northeast-1.compute.internal:postgresql (ESTABLISHED)
uvicorn    1             root   14u     IPv4           18369763      0t0      TCP 59c08b6aac89:56040->ip-172-31-5-149.ap-northeast-1.compute.internal:postgresql (ESTABLISHED)
uvicorn    1             root   15u     IPv4           18369373      0t0      TCP 59c08b6aac89:56042->ip-172-31-5-149.ap-northeast-1.compute.internal:postgresql (ESTABLISHED)
uvicorn    1             root   16u     IPv4           18370707      0t0      TCP 59c08b6aac89:56044->ip-172-31-5-149.ap-northeast-1.compute.internal:postgresql (ESTABLISHED)
uvicorn    1             root   17u     IPv4           18402915      0t0      TCP 59c08b6aac89:https->61-30-51-61.static.tfn.net.tw:26547 (ESTABLISHED)
uvicorn    1             root   18u     IPv4           18373916      0t0      TCP 59c08b6aac89:https->61-222-56-55.HINET-IP.hinet.net:14034 (ESTABLISHED)
uvicorn    1             root   19u     IPv4           18479991      0t0      TCP 59c08b6aac89:https->125-230-66-32.dynamic-ip.hinet.net:46420 (ESTABLISHED)
uvicorn    1             root   20u     IPv4           18604769      0t0      TCP 59c08b6aac89:https->125-227-151-121.HINET-IP.hinet.net:61864 (ESTABLISHED)
uvicorn    1             root   21u     IPv4           18509169      0t0      TCP 59c08b6aac89:https->114-137-84-64.emome-ip.hinet.net:54518 (ESTABLISHED)
uvicorn    1             root   22u     IPv4           18668143      0t0      TCP 59c08b6aac89:https->125-227-151-121.HINET-IP.hinet.net:63229 (ESTABLISHED)
uvicorn    1             root   23u     IPv4           18520950      0t0      TCP 59c08b6aac89:https->125-227-151-121.HINET-IP.hinet.net:60139 (ESTABLISHED)
uvicorn    1             root   24u     IPv4           18374580      0t0      TCP 59c08b6aac89:https->211-75-187-47.HINET-IP.hinet.net:56651 (ESTABLISHED)
uvicorn    1             root   25u     IPv4           18376897      0t0      TCP 59c08b6aac89:https->61-222-56-55.HINET-IP.hinet.net:14167 (ESTABLISHED)
uvicorn    1             root   26u     IPv4           18386427      0t0      TCP 59c08b6aac89:https->210.241.98.253:12898 (ESTABLISHED)
uvicorn    1             root   27u     IPv4           18399495      0t0      TCP 59c08b6aac89:https->101.10.59.24:19382 (ESTABLISHED)
uvicorn    1             root   28u     IPv4           18409532      0t0      TCP 59c08b6aac89:https->36-224-32-223.dynamic-ip.hinet.net:1918 (ESTABLISHED)
uvicorn    1             root   29u     IPv4           18386429      0t0      TCP 59c08b6aac89:https->210.241.98.253:12899 (ESTABLISHED)
uvicorn    1             root   30u     IPv4           18388350      0t0      TCP 59c08b6aac89:https->210.241.98.253:12900 (ESTABLISHED)
uvicorn    1             root   31u     IPv4           18388352      0t0      TCP 59c08b6aac89:https->210.241.98.253:12901 (ESTABLISHED)
uvicorn    1             root   32u     IPv4           18494758      0t0      TCP 59c08b6aac89:https->125-230-66-32.dynamic-ip.hinet.net:47366 (ESTABLISHED)
uvicorn    1             root   33u     IPv4           18372777      0t0      TCP 59c08b6aac89:56334->ip-172-31-5-149.ap-northeast-1.compute.internal:postgresql (ESTABLISHED)
uvicorn    1             root   34u     IPv4           18386431      0t0      TCP 59c08b6aac89:https->61-222-56-55.HINET-IP.hinet.net:14590 (ESTABLISHED)
uvicorn    1             root   35u     IPv4           18370173      0t0      TCP 59c08b6aac89:https->61-222-56-55.HINET-IP.hinet.net:13950 (ESTABLISHED)
uvicorn    1             root   36u     IPv4           18372577      0t0      TCP 59c08b6aac89:https->61-222-56-55.HINET-IP.hinet.net:13988 (ESTABLISHED)
uvicorn    1             root   37u     IPv4           18530325      0t0      TCP 59c08b6aac89:https->49.216.39.2:8508 (ESTABLISHED)
uvicorn    1             root   38u     IPv4           18370760      0t0      TCP 59c08b6aac89:56102->ip-172-31-5-149.ap-northeast-1.compute.internal:postgresql (ESTABLISHED)
uvicorn    1             root   39u     IPv4           18388354      0t0      TCP 59c08b6aac89:https->210.241.98.253:12902 (ESTABLISHED)
uvicorn    1             root   40u     IPv4           18554686      0t0      TCP 59c08b6aac89:https->210-209-175-100.veetime.com:9852 (ESTABLISHED)
uvicorn    1             root   41u     IPv4           18441399      0t0      TCP 59c08b6aac89:https->36-227-149-58.dynamic-ip.hinet.net:63682 (ESTABLISHED)
...
...
...
...

There shows like 500u or more after server up time for 24h

When it hit the limit, the server no longer can accept any new request.
docker logs shows below:

ERROR:asyncio:socket.accept() out of system resource
socket: <asyncio.TransportSocket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('0.0.0.0', 443)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 164, in _accept_connection
    conn, addr = sock.accept()
  File "/usr/local/lib/python3.9/socket.py", line 293, in accept
    fd, addr = self._accept()
OSError: [Errno 24] Too many open files

I have to down and up the docker container again.
And the number will go down to only like 10u

Run ulimit -a in container:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 63280
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 63280
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

I already tried increase the open files (-n) 1024.
But it didn’t resolve the problem.
Only increase the time to happen this problem.

Environment

Run server by following command:

uvicorn main:app --host 0.0.0.0 --port 443 --proxy-headers --timeout-keep-alive 60 --limit-concurrency 1000 --ssl-keyfile=./xxxx.key --ssl-certfile=./xxxx.cer

uvicorn --version:

Running uvicorn 0.13.4 with CPython 3.9.4 on Linux

python --version:

cat /etc/os-release in container:

PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

@jeho0815

the open files is 1024, you should increase it!
open files (-n) 1024

@SynclabIO

the open files is 1024, you should increase it!
open files (-n) 1024

I already tried it. It only increase time to this error.
It still happen after more time.
It shouldn’t happen.
Even I set up firewall to block all connection.
The lsof number is still not going down.

@euri10

Hard to say if this is on our side (there’s nothing in the logs that says something about uvicorn or I’m missing something ?).

It would be also interesting to check if this happens without ssl enabled.

If that does happen only after 24h it’s going to be hard to reproduce so please try to come up with a reproducible example that would trigger that without being that long so we can investigate.

On my side I tried to reproduce with an app running pretty much like yours uvicorn app:app --proxy-headers --timeout-keep-alive 60 --ssl-keyfile=server.key --ssl-certfile=server.pem
Hitting it with wrk for 3 minutes with wrk -c 2048 -d 3m -t16 https://localhost:8000
I run also an lsof check with watch -n 1 'lsof -i :8000 | wc -l' and it’s stable at 4098 for the whole 3 minutes so no issues nor ESTABLISHED increasing over time, i see no reason why this would occur after a given period of time, we should be able to see this asap.

@tkor1

I see this problem too. Some of my experience:

  1. lsof shows ~800 before starting the server and goes to about 880 upon start. netstat shows ~120 lines (all included, even headers).
  2. Not an active server, I’m playing on the side with it, so no load except my control test.
  3. I see similar result for the open file limit set to 900, 1024 and 2048.
  4. I start the server and immediately do an HTTP request, getting response successfully.
  5. Leave the server running for about 10-15minutes without any HTTP request. lsof stays pretty stable at around 880. netstat shows ~120 lines (all included, even headers).
  6. Check lsof and netstat and you see the same numbers. Make an HTTP request and it doesn’t get a response, the server log shows the asyncio message that @SynclabIO posted. The error message only happens upon making the new HTTP request. no change on lsof and netstat.
  7. At this point I couldn’t find any remedy to make it work again. If I move the task to background, change the ulimit and move back to the foreground it just Sigfault.

From what I can tell, some resource is leaking, but I can’t point to the specific one.

@euri10

what would help is the application or the endpoint your’re suspecting that would leak.

@euri10

will close this as stale, feel free to reopen if the issue persists

@slava-shor-emporus

Hm… It looks like we experienced precisely the same issue. But in our case takes a few days until the service enters into a loop of such error without a recovery.

asyncio - ERROR - socket.accept() out of system resource
socket: <asyncio.TransportSocket fd=9, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('0.0.0.0', 8080)>
Traceback (most recent call last):
File "/usr/local/lib/python3.10/asyncio/selector_events.py", line 159, in _accept_connection
conn, addr = sock.accept()
File "/usr/local/lib/python3.10/socket.py", line 293, in accept
fd, addr = self._accept()
OSError: [Errno 24] Too many open files

@AranVinkItility

Same here, running uvicorn 0.20.0/FastAPI 0.89.1/Python 3.9 inside a container on ECS. Ran fine for a few days with some occasional load and then errors out.

ERROR:asyncio:socket.accept() out of system resource
socket: <asyncio.TransportSocket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 80)>
Traceback (most recent call last):
File "/venv/lib/python3.9/asyncio/selector_events.py", line 159, in _accept_connection
File "/venv/lib/python3.9/socket.py", line 293, in accept
OSError: [Errno 24] Too many open files

@slava-shor-emporus

Same here, running uvicorn 0.20.0/FastAPI 0.89.1/Python 3.9 inside a container on ECS. Ran fine for a few days with some occasional load and then errors out.

Pardon me, forgot to attach our spec.
We run on AWS Fargate in a docker image build from python:3.10-slim (current Python version 3.10.2) and among dependencies:

  • fastapi 0.89.1
  • uvicorn 0.20.0
  • uvloop 0.17.0

@slava-shor-emporus

I sense @AranVinkItility we have something in common. We are running containers on AWS.

@DazEdword

I am having a similar issue. I am just experimenting with FastAPI for the first time, so my application is barely a hello world. It seems to run OK locally (directly), but when trying to run once it’s containerised it throws this error right away:

Skipping virtualenv creation, as specified in config file.
INFO:     Will watch for changes in these directories: ['/app']
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [1] using WatchFiles
Traceback (most recent call last):
  File "/usr/local/bin/uvicorn", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/uvicorn/main.py", line 404, in main
    run(
  File "/usr/local/lib/python3.10/site-packages/uvicorn/main.py", line 564, in run
    ChangeReload(config, target=server.run, sockets=[sock]).run()
  File "/usr/local/lib/python3.10/site-packages/uvicorn/supervisors/basereload.py", line 45, in run
    for changes in self:
  File "/usr/local/lib/python3.10/site-packages/uvicorn/supervisors/basereload.py", line 64, in __next__
    return self.should_restart()
  File "/usr/local/lib/python3.10/site-packages/uvicorn/supervisors/watchfilesreload.py", line 85, in should_restart
    changes = next(self.watcher)
  File "/usr/local/lib/python3.10/site-packages/watchfiles/main.py", line 119, in watch
    with RustNotify([str(p) for p in paths], debug, force_polling, poll_delay_ms, recursive) as watcher:
_rust_notify.WatchfilesRustInternalError: Error creating recommended watcher: Too many open files (os error 24)

My dependencies (using Poetry):

[tool.poetry.dependencies]
python = "^3.10"
fastapi = "^0.89.1"
uvicorn = {extras = ["standard"], version = "^0.20.0"}

The docker image causing trouble is python:3.10-slim-bullseye. Trying to increase the open files in the container doesn’t seem to help.

Edit: After some further experimentation with open file limits I have managed to make it work, which suggests there is an issue with docker and the local system defining their ulimit.

Setting Docker ulimit explicitly in the docker commands did not work:
docker build . -t fastapi-template && docker run --rm --ulimit nofile=90000:90000 -it -p 8000:8000 fastapi-template

However, bumping my local user’s ulimit as explained here helped: https://stackoverflow.com/a/24331638
Then I could run:

docker build . -t fastapi-template && docker run --rm -it -p 8000:8000 fastapi-template

@zvolsky

Same here,
pyenv local 3.11.2
poetry env use $(which python)
fastapi
and HelloWorld application.
uvicorn subdir.main:app –reload –reload-include=’subdir/’

[tool.poetry.dependencies]
python = “^3.11”
fastapi = “^0.94.1”
uvicorn = {extras = [“standard”], version = “^0.21.0”}
tortoise-orm = “^0.19.3”
aerich = “^0.7.1”
python-multipart = “^0.0.6”
watchfiles = “^0.18.1”

@zvolsky

Regarding DazEdword and my similar issue it looks like that it is bound to the usage of poetry. If I recreate same virtual environment using python -m venv and pip, then I have no problem.
So maybe some interaction of Rust watchfiles notifier and poetry. Maybe the error is internally (rust?) something completly different and just on python level raised as “Too many files”?
I don’t understand the mechanism of raising this python exception and I don’t know how to see the original error from rust.

This was referenced

Mar 15, 2023

I am running a Go binary that has an http server with systemd. I have it setup so that Caddy has a reverse-proxy to this go http server.

http: Accept error: accept tcp [::]:8002: accept4: too many open files;
dial tcp 192.85.2.4:443: socket: too many open files

When I look at the open files of the process, I get 1025 or less, though my ulimit is set to a much larger limit:

$ lsof -p 1243 | wc -l
1025
$ ulimit -Sn
200000
$ ulimit -Hn
1048576

I’m not sure if that’s the problem but seems that it could be? Seems like the Go server should be spawning new goroutines or take care of that somehow.

EDIT: Here’s my server script:

package main
  
import (
        "fmt"
        "time"
        "net/http"
)

type Config struct{}


func (c *Config) testerHandler(w http.ResponseWriter, r *http.Request) {
    r.Body.Close()
    time.Sleep(1 * time.Second)
    fmt.Fprintf(w, "hello\n")
}

func main() {
        c := &Config{}
        http.HandleFunc("/tester", c.testerHandler)
        fmt.Println("listening on http://127.0.0.1:8000")
        http.ListenAndServe(":8000", nil)
}

EDIT: And here’s a script I use to spam my server:

package main
  
import (
        "log"
        "net/http"
)

func main() {
        number := 10000
        log.Printf("spamming %d numbers\n", number)
        ch := make(chan interface{})
        client := http.Client{}

        for i:=0; i<number; i++{
                go func(number int) {
                        u := "http://127.0.0.1:8000/tester"
                        rsp, err := client.Get(u)
                        if err != nil {
                                ch <- err
                                return
                        }
                        rsp.Body.Close()

                        ch <- rsp
                }(number)
        }

        var errs int
        m := make(map[int]int)

        for i:=0; i<number; i++ {
                rsp := <-ch
                switch rsp.(type) {
                case *http.Response:
                        code := rsp.(*http.Response).StatusCode
                        m[code]++
                default:
                        log.Println(rsp.(error))
                        errs++
                }
        }

        log.Println("errs:", errs)
        for k, v := range m {
                log.Printf("%d:%d\n", k, v)
        }
}

Добавить комментарий

Ваш адрес email не будет опубликован. Обязательные поля помечены *