Abstract Internet is vulnerable to overloading caused by flash crowds and flooding denial-of-service (DoS) attacks. Recently voice over IP (VoIP), an Internet-based service is experiencing a phenomenal growth. As its deployment spreads, VoIP systems are likely to become attack targets, of which flooding lists high, perhaps due to its simplicity and the abundance of tool support. In this study, we are particularly interested in detecting and distinguishing INVITE flooding DoS attack from an INVITE surge due to a flash crowd event. Both of these events degrade the performance of a SIP proxy server to the point where it becomes sluggish and even unresponsive. Therefore, an early detection and a way to distinguish between INVITE flooding and flash crowd is an acute contemporary problem. Accurate early detection makes it possible to tackle INVITE requests surge either by dropping malicious requests or by regulating legitimate requests before it can deteriorate the normal service of a SIP proxy server. As a detection mechanism, our proposed solution characterizes normal SIP protocol behavior and anomalies thereof, using information theoretic approach based on the Hellinger distance, which computes the variability between two probability distributions. We develop a heuristic to detect anomalous protocol behavior using Chebyshev inequality and the distribution of computational distance values. Additionally, we present a fundamentally different and practical approach for server-side overloading protection. Our approach exploits protocol's reliability mechanism and its timer values to regulate and recognize legitimate requests from the attack traffic.