So, if I do some benchmarking with apache benchmark (ab), and I use large numbers of requests. Then sometimes in the middle of a test I get this error.

I don’t even know what it means. So how can I fix it? Or is it just something that will happen if the server gets too many hits anyway? The problem is, if I run 10,000 hits, it’ll all run perfectly. If I run it again, it’ll get to 4000 and get the error:

apr_socket_recv: Connection reset by peer (104)

A little about my setup:
I have nginx taking static requests and processing dynamic ones to apache. The file in question is served from cache by nginx, so I guess it’s probably got to do with how nginx is handling the requests?

Ideas?

asked May 30, 2010 at 0:59

Matthew's user avatar

The error means that the other end (webserver) suddenly disconnected in the middle of the session.
have a look at the apache or nginx error logs to see if there is anything suspicious there.

answered May 30, 2010 at 7:43

Aleksandar Ivanisevic's user avatar

It means that server is heavly loaded with the request i.e, all the threads are busy serving the request.
Solution : either increase the maxThread attribute count for connector in server.xml file or increase acceptCount attribute value.

acceptcount : The maximum queue length for incoming connection requests when all possible request processing threads are in use. Any requests received when the queue is full will be refused.

answered Mar 13, 2014 at 7:21

Kushal Bafna's user avatar

Besides the answers here, I have read a lot of other ones:

  • Replace localhost by 127.0.0.1
  • Update the apache version (I have ApacheBench, Version 2.3 <$Revision: 1807734 $>)
  • Add -r (Then I get apr_pollset_poll: The timeout specified has expired (70007))

None of them helped.

I thought about switching to wrk after seeing similar struggles.

Finding the problem

The problem seems to be related to the amount of ephermal ports. I tried to set it from 50000 to 25000 as this is the port range. Still no luck. Then I got the impression it is related to TIME_WAIT and this blog post. I think I could confirm that:

$ netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n

    1 CLOSE_WAIT
    1 established)
    1 Foreign
    4 LISTEN
    8 SYN_SENT
   62 SYN_RECV
  351 ESTABLISHED
13916 TIME_WAIT

What I tried

I didn’t fix it so far :-/

According to sudo sysctl -a | grep net.ipv4.tcp, I have:

net.ipv4.tcp_tw_reuse = 0    # No luck setting only that to 1
net.ipv4.tcp_max_tw_buckets = 32768
net.ipv4.tcp_fin_timeout = 60  # Setting it to 5 didn't help either

answered Jul 11, 2018 at 21:24

Martin Thoma's user avatar

This issue is caused by the system. if give a high concurrency request to the system. OS kernel will trigger SYN flood protection. So the system will reset the link.
you can modify the OS config in the file.

#vi /etc/sysctl.conf
net.ipv4.tcp_syncookies = 0 # set value is 0
#sysctl -p # read config from the config file.

you can try it.

usually the attribute net.ipv4.tcp_syncookies was used to protect OS to avoid the huge request attack. But if you want to use this OS to do some Load test or performance test, you should close this feature.

Jenny D's user avatar

Jenny D

27.9k21 gold badges75 silver badges114 bronze badges

answered Nov 30, 2017 at 22:46

moneyfly's user avatar

I had same problem and my server version was:

Server Version: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 PHP/5.6.5 mod_perl/2.0.9dev Perl/v5.16.3

i removed unneccessary modules and problem is gone:

Server Version: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips

So one of mod_fcgid, mod_php or mod_perl is causing problem. You can try to disable those if you are not using.

(Side note;
If you are using opcache,disable fast_shutdown too. It was causing problem too:
opcache.fast_shutdown=0 )

answered Feb 23, 2015 at 6:30

Ünsal Korkmaz's user avatar

0

You must log in to answer this question.

Not the answer you’re looking for? Browse other questions tagged

.

So, if I do some benchmarking with apache benchmark (ab), and I use large numbers of requests. Then sometimes in the middle of a test I get this error.

I don’t even know what it means. So how can I fix it? Or is it just something that will happen if the server gets too many hits anyway? The problem is, if I run 10,000 hits, it’ll all run perfectly. If I run it again, it’ll get to 4000 and get the error:

apr_socket_recv: Connection reset by peer (104)

A little about my setup:
I have nginx taking static requests and processing dynamic ones to apache. The file in question is served from cache by nginx, so I guess it’s probably got to do with how nginx is handling the requests?

Ideas?

asked May 30, 2010 at 0:59

Matthew's user avatar

The error means that the other end (webserver) suddenly disconnected in the middle of the session.
have a look at the apache or nginx error logs to see if there is anything suspicious there.

answered May 30, 2010 at 7:43

Aleksandar Ivanisevic's user avatar

It means that server is heavly loaded with the request i.e, all the threads are busy serving the request.
Solution : either increase the maxThread attribute count for connector in server.xml file or increase acceptCount attribute value.

acceptcount : The maximum queue length for incoming connection requests when all possible request processing threads are in use. Any requests received when the queue is full will be refused.

answered Mar 13, 2014 at 7:21

Kushal Bafna's user avatar

Besides the answers here, I have read a lot of other ones:

  • Replace localhost by 127.0.0.1
  • Update the apache version (I have ApacheBench, Version 2.3 <$Revision: 1807734 $>)
  • Add -r (Then I get apr_pollset_poll: The timeout specified has expired (70007))

None of them helped.

I thought about switching to wrk after seeing similar struggles.

Finding the problem

The problem seems to be related to the amount of ephermal ports. I tried to set it from 50000 to 25000 as this is the port range. Still no luck. Then I got the impression it is related to TIME_WAIT and this blog post. I think I could confirm that:

$ netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n

    1 CLOSE_WAIT
    1 established)
    1 Foreign
    4 LISTEN
    8 SYN_SENT
   62 SYN_RECV
  351 ESTABLISHED
13916 TIME_WAIT

What I tried

I didn’t fix it so far :-/

According to sudo sysctl -a | grep net.ipv4.tcp, I have:

net.ipv4.tcp_tw_reuse = 0    # No luck setting only that to 1
net.ipv4.tcp_max_tw_buckets = 32768
net.ipv4.tcp_fin_timeout = 60  # Setting it to 5 didn't help either

answered Jul 11, 2018 at 21:24

Martin Thoma's user avatar

This issue is caused by the system. if give a high concurrency request to the system. OS kernel will trigger SYN flood protection. So the system will reset the link.
you can modify the OS config in the file.

#vi /etc/sysctl.conf
net.ipv4.tcp_syncookies = 0 # set value is 0
#sysctl -p # read config from the config file.

you can try it.

usually the attribute net.ipv4.tcp_syncookies was used to protect OS to avoid the huge request attack. But if you want to use this OS to do some Load test or performance test, you should close this feature.

Jenny D's user avatar

Jenny D

27.9k21 gold badges75 silver badges114 bronze badges

answered Nov 30, 2017 at 22:46

moneyfly's user avatar

I had same problem and my server version was:

Server Version: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 PHP/5.6.5 mod_perl/2.0.9dev Perl/v5.16.3

i removed unneccessary modules and problem is gone:

Server Version: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips

So one of mod_fcgid, mod_php or mod_perl is causing problem. You can try to disable those if you are not using.

(Side note;
If you are using opcache,disable fast_shutdown too. It was causing problem too:
opcache.fast_shutdown=0 )

answered Feb 23, 2015 at 6:30

Ünsal Korkmaz's user avatar

0

You must log in to answer this question.

Not the answer you’re looking for? Browse other questions tagged

.

OS: Wheezy
bitcoind 0.9.1 pre-build
libminiupnp 1.6-3 from testing

Since the update to 0.9.1 i cant get blocks but a log full of that 104 error.

2014-04-19 07:45:27 ProcessBlock: ACCEPTED
2014-04-19 07:46:46 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=[censored]:8333, them=0.0.0.0:0, peer=[2a02:348:5e:5a29::1]:48263
2014-04-19 07:46:58 receive version message: /getaddr.bitnodes.io:0.1/: version 70001, blocks=296603, us=[censored]:8333, them=0.0.0.0:0, peer=[2a01:4f8:131:167::2]:13711
2014-04-19 07:46:58 socket recv error 104
2014-04-19 07:51:15 receive version message: /getaddr.bitnodes.io:0.1/: version 70001, blocks=296603, us=[censored]:8333, them=0.0.0.0:0, peer=[2a01:4f8:131:167::2]:11525
2014-04-19 07:51:16 socket recv error 104
2014-04-19 07:54:08 receive version message: /getaddr.bitnodes.io:0.1/: version 70001, blocks=296603, us=[censored]:8333, them=0.0.0.0:0, peer=[2a01:4f8:131:167::2]:45595
2014-04-19 07:54:08 socket recv error 104
2014-04-19 07:54:40 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=[censored]:8333, them=0.0.0.0:0, peer=[2001:470:88ff:2e::1]:35029
2014-04-19 07:57:14 receive version message: /getaddr.bitnodes.io:0.1/: version 70001, blocks=296603, us=[censored]:8333, them=0.0.0.0:0, peer=[2a01:4f8:131:167::2]:28512
2014-04-19 07:57:21 socket recv error 104
2014-04-19 07:58:18 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
2014-04-19 07:58:22 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
2014-04-19 07:58:25 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
2014-04-19 07:58:27 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
2014-04-19 07:58:31 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
2014-04-19 07:58:33 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
2014-04-19 07:58:39 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
2014-04-19 07:58:41 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
2014-04-19 07:58:43 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
2014-04-19 07:58:44 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
2014-04-19 07:58:46 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
2014-04-19 07:58:47 ERROR: AcceptToMemoryPool : nonstandard transaction: dust
2014-04-19 08:02:29 receive version message: /getaddr.bitnodes.io:0.1/: version 70001, blocks=296603, us=[censored]:8333, them=0.0.0.0:0, peer=[2a01:4f8:131:167::2]:40801
2014-04-19 08:02:30 socket recv error 104
2014-04-19 08:03:46 receive version message: /getaddr.bitnodes.io:0.1/: version 70001, blocks=296603, us=[censored]:8333, them=0.0.0.0:0, peer=[2a01:4f8:131:167::2]:58055
2014-04-19 08:03:48 socket recv error 104
2014-04-19 08:04:19 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=[censored]:8333, them=0.0.0.0:0, peer=[2a02:348:5e:5a29::1]:43272
2014-04-19 08:06:10 ProcessBlock: ACCEPTED
2014-04-19 08:08:34 receive version message: /getaddr.bitnodes.io:0.1/: version 70001, blocks=296604, us=[censored]:8333, them=0.0.0.0:0, peer=[2a01:4f8:131:167::2]:59327
2014-04-19 08:08:34 socket recv error 104
2014-04-19 08:10:16 keypool reserve 5
2014-04-19 08:10:16 keypool return 5
2014-04-19 08:11:36 receive version message: /getaddr.bitnodes.io:0.1/: version 70001, blocks=296604, us=[censored]:8333, them=0.0.0.0:0, peer=[2a01:4f8:131:167::2]:39861
2014-04-19 08:11:37 socket recv error 104
2014-04-19 08:14:14 receive version message: /getaddr.bitnodes.io:0.1/: version 70001, blocks=296604, us=[censored]:8333, them=0.0.0.0:0, peer=[2a01:4f8:131:167::2]:14525
2014-04-19 08:14:15 socket recv error 104
2014-04-19 08:17:07 ProcessBlock: ACCEPTED
2014-04-19 08:19:18 receive version message: /getaddr.bitnodes.io:0.1/: version 70001, blocks=296605, us=[censored]:8333, them=0.0.0.0:0, peer=[2a01:4f8:131:167::2]:24259
2014-04-19 08:19:20 socket recv error 104
2014-04-19 08:21:57 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=[censored]:8333, them=0.0.0.0:0, peer=[2a02:348:5e:5a29::1]:38561
2014-04-19 08:22:19 receive version message: /getaddr.bitnodes.io:0.1/: version 70001, blocks=296605, us=[censored]:8333, them=0.0.0.0:0, peer=[2a01:4f8:131:167::2]:64099
2014-04-19 08:22:20 socket recv error 104

sorry i’m very new to bitcoind but this never happend before.

Zcash Community Forum

Loading

I have a code where there are 13 clients that have to connect to the server. Then the server does some counting on the data given by the client. After that the roles turns around – the server becomes a client and clients become servers to receive the data.
The thing is that when trying to do the first connection, that is when the 13 clients try to connect to the server I keep getting this error: [Errno 104] Connection reset by peer. I tried some workarounds for example trying to connect 5 times in a second interval but nothing works.

Here my code:

server.py

import socket, pickle, numpy as np
import struct
import math


while 1:
    HOST = ''
    PORT = 50007
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.bind((HOST, PORT))
    s.listen(13)

adresses = []
ports = []

i = 0
print("receiving...")
while i < 13:
    i += 1    
    #wait to accept a connection - blocking call
    conn, addr = s.accept()
    print ('Connected with ', addr)
    adresses.append(addr[0])
    buf = b''
    while len(buf) < 4:
        buf += conn.recv(4 - len(buf))
    length = struct.unpack('>I', buf)[0]
    data = b''
    l = length

    while l > 0:

        d = conn.recv(l)
        l -= len(d)
        data += d

    if not data: break

    M = np.loads(data)

    if i == 1:
        L = M[0]
    else:
        L += M[0]
    ports.append(M[1])
    conn.close() 
s.close()


L /= 993040

packet = pickle.dumps(L)
length = struct.pack('>I', len(packet))
packet = length + packet

print("sending...")
for kl, addr in enumerate(adresses):
    HOST = addr
    PORT = 50007 + ports[kl]
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.connect((HOST, PORT))
    s.sendall(packet)
    s.close()

client.py

def connection(centers, kl):

    HOST = "192.168.143.XX"
    PORT = 50007
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.settimeout(3600)
    try:
        s.connect((HOST, PORT)) # HERE IS AN ERROR
        s.settimeout(None)
        packet = pickle.dumps([centers, kl]) ## ???
        length = struct.pack('>I', len(packet))
        packet = length + packet
        s.sendall(packet) # OR HERE IS AN ERROR
        s.close()
    except Exception as e:
        print(e)
        print('error ', kl)
        s.close()
        return np.zeros(centers.shape)    


    HOST = ''
    PORT = 50007 + kl
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.bind((HOST, PORT))

    s.listen(2)

    i = 0
    while i < 1:
        #wait to accept a connection - blocking call
        conn, addr = s.accept()
        i += 1
        print ('Connected with ', addr)
        buf = b''
        while len(buf) < 4:
            buf += conn.recv(4 - len(buf))

        length = struct.unpack('>I', buf)[0]
        data = b''
        l = length
        while l > 0:
            d = conn.recv(l)
            l -= len(d)
            data += d
        if not data: break

        new_centers = np.loads(data)
        conn.close()

    s.close()
    return new_centers

aa = 0
for k in range(99):
    print(k)
    centers = some_function(centers)
    time.sleep(60)
    centers1 = connection(centers, i)
    aa = 0
    while not (centers1.any()) and aa < 5:
        time.sleep(1)
        centers1 = connection(centers, i)
        aa += 1
    centers = centers1

The thing is all of the 13 clients HAVE TO connect to the server or it won’t proceed to the next iteration.
I’m using Python 3.4.
Please help.

Update:

I have added threads but the error remains:

[Errno 104] Connection reset by peer

server.py

import socket, pickle, numpy as np
import struct
import math
from multiprocessing.pool import ThreadPool

def clientthread(conn, L):
    buf = b''
    while len(buf) < 4:
        buf += conn.recv(4 - len(buf))
    length = struct.unpack('>I', buf)[0]
    data = b''
    l = length
    while l > 0:
        d = conn.recv(l)
        l -= len(d)
        data += d
    M = np.loads(data)


    return(M)


j = 0
while 1:
    HOST = ''
    PORT = 50007
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.bind((HOST, PORT))
    s.listen(2)
    #print('0')

    adresses = []
    ports = []

    i = 0
    print("receiving...")
    while i < 13:
        i += 1    
        #wait to accept a connection - blocking call
        conn, addr = s.accept()
        print ('Connected with ', addr)
        adresses.append(addr[0])
        pool = ThreadPool(processes=13)
        async_result = pool.apply_async(clientthread, (conn, i,))
        M = async_result.get()  
        conn.close() 

        if i == 1:
            L = M[0]
        else:
            L += M[0]
        ports.append(M[1])
    s.close()


    L /= 993040

    packet = pickle.dumps(L)
    length = struct.pack('>I', len(packet))
    packet = length + packet

    for kl, addr in enumerate(adresses):
        HOST = addr
        PORT = 50007 + ports[kl]
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.connect((HOST, PORT))
        s.sendall(packet)
        s.close()

Добавить комментарий

Ваш адрес email не будет опубликован. Обязательные поля помечены *